mirror of
https://github.com/OPCFoundation/UA-.NETStandard.git
synced 2026-10-16 21:02:51 +08:00
cef83550cb
# Description Implements **OPC UA — Vision** and uses it to build the demo the companion specifications were drafted for: an eye-in-hand camera watches a bin of parts, a language model *looks at the frame*, decides which part matches the instruction, and commands the robot through Robot Intent — with the OpenUSD viewer showing the arm move and the bin empty out. > **Draft, and stacked.** [#4195](https://github.com/OPCFoundation/UA-.NETStandard/pull/4195) has merged and master is merged in, so what is left on top of `master` is the Vision work plus the unmerged `marcschier/ai-model-management` branch, which the demo needs for the model/dataset/deployment provenance. Review the Vision commits; the AI Model Management work lands with its own PR. ## What is here **The specification, implemented.** `Opc.Ua.Vision` is generated from the spec's own `Opc.Ua.Vision.NodeSet2.xml`, vendored byte-identically (SHA-256 `79C592C6…`, verified against a fresh download and stored LF-preserved like the Robot Intent NodeSet). `Opc.Ua.Vision.Server` builds the §4.2 discovery object, sensors, the coordinate-frame tree, calibrations, media endpoints, inference pipelines and the §9 feedback methods, with rendering and inference behind injectable providers. `Opc.Ua.Vision.Client` is built on the generated ObjectType proxies and composes transforms across the frame tree, which is what turns a detection in camera coordinates into a pose a robot can act on. **Two perception paths behind one contract**, which is exactly what `InferenceLocationEnum` is for. `OnServer` is a deterministic detector derived from the stage's ground truth — no model, no network, no GPU, so CI can exercise the whole loop. `EdgeOffServer` is the agent: it sees the frame over MCP and calls `SubmitDetections`, and the Server publishes results it did not compute. A client reads `DetectionResultType` identically either way. **The agent's eyes.** `tools/Opc.Ua.Mcp.Vision` adds 22 `vision_*` tools, which with the four shared connection tools makes the `vision` profile 26; `vision_get_frame` returns the camera image as an MCP `ImageContentBlock`, so the model looks at pixels rather than reading a description of them. Tool profiles now compose — `--profile vision,robotics` yields 62 tools — so one agent can both see and act without dragging in the full 136-tool catalogue. **The cell.** `samples/Robotics/BinPickingCell` follows the spec's own Robotics-Vision Addendum: the `world → robot_base → flange → gripper_tcp` frame tree with `camera_eih` on the flange, the `HandEye` extrinsics and the `Intrinsics612x512` intrinsics — the addendum's 2448×2048 calibration scaled to the 4×4-binned grid the camera actually delivers. `samples/Robotics/BinPickingClient` hosts the composed MCP catalogue and optionally the viewer. **Documentation.** `docs/Vision.md` (~800 lines) covers the model, hosting, the build context, every topology builder, both perception paths, feedback, the MCP profile and the limitations; linked from `docs/README.md`, with the profile table in `docs/McpServer.md` and READMEs for both samples. ## Evidence, not assertions Each of these was measured on a running system rather than assumed: | Claim | Evidence | |---|---| | The camera sees the bin | 612×512 frame with the bin and all five parts in view — red `(212,47,47)`, green `(61,193,71)`, blue `(54,85,218)`, yellow `(215,196,58)`, orange `(209,119,55)` | | An agent can act on what it sees | Over MCP, `vision_get_frame` returns a 612×512 PNG that decodes from base64 to 1,254,051 bytes, the descriptor beside the detections says 612×512, and all five `BoundingBox2D` centres land **inside** the frame on the pixel colour their own detection claims | | The geometry is right | A detection composed `camera_eih → world` lands on the authored position, residual **0.0000 m** — on both the on-server and off-server paths | | The detector tracks the world | Placing a part drops the next scan from 5 detections to 4, with that part absent | | The robot obeys the agent | Scripted run took command authority and completed a pick and a place, both `Succeeded` | | Tool catalogues are honest | Counted from a running server: vision 26, robotics 40, `vision,robotics` 62, full 136, **zero duplicates** | | Every node is browsable by NodeId | Eight representative NodeIds — both folders, a pipeline, a frame, the sensor, its Calibrations folder, the HandEye calibration, the Media folder — all `Good` | That tool-count check matters because the profile tests collapse names into a `HashSet` and *cannot see* a duplicate registration — so the new tests count registrations, not names. ## Things that were wrong, and are not now Writing the tests found three defects of one kind — code that answered instead of refusing. A zero-norm quaternion was silently rewritten to identity, so a caller with an all-zero orientation got a confident wrong grasp pose. `SubmitCorrection` rewrote a null `ResultId` to empty and called the sink anyway. A bad `StageIdentifier` **killed the host process** inside native `UsdStage.Open`, via an exception .NET cannot catch. All three now refuse loudly and are pinned by tests. **Nodes that existed but could not be reached.** The node manager indexes the Vision root before configurators run, so everything the fluent builder grafted on afterwards — both folders, the pipelines with their Results and Feedback, the frames, the sensors with their optics, calibrations and media endpoints — was never added to the index. Browsing *forward from a parent* worked, because that walks `NodeState.Children` in memory; browsing any of those nodes *by its own NodeId* returned `BadNodeIdUnknown`, which is how an ordinary client and the MCP discovery tools navigate. Registration is now deferred and flushed after each configurator. The first fix covered only the DI path and left the documented non-DI fallback broken in exactly the same way, so `ConfigureVisionAsync` now exists and registers what it builds. That defect survived a green suite because every test browsed children. The new tests assert by NodeId, and were checked against the broken code — with the flush removed, they fail. **Every Method was uncallable.** The Server passed 426 unit tests while not one of its Methods could be invoked over a real session — `RunInference` returned `Bad_TooManyArguments`, and so did every other Method that takes arguments, which is all of them but `StartContinuous` and `Stop`. Three independent faults, each sufficient on its own: the Method nodes were never created (they are Optional children, and the dispatcher guards every attachment with a null check, so it silently attached nothing); nothing declared the arguments, so the stack concluded none were expected; and `MethodDeclarationId` pointed at a synthesised NodeId rather than the NodeSet declaration a client calls with, so the lookup missed the instance Method entirely. The builder now creates the Methods, the `Results` folder and the `Feedback` object when a provider or sink is configured, declares every signature, and names the right declaration. Unit tests could not have caught this: they invoke the handler delegates directly and never go through `MethodState.Call`. It took an end-to-end test over a real session. Relatedly, children created by the generated `CreateOrReplace` helpers carry no `ReferenceTypeId`, so nothing could reach them — fixed once in the build context rather than at the several dozen call sites. **And one where I was wrong.** Earlier in this branch I reversed my own agents' decision and made `SubmitDetections` accept an empty detection set, reasoning that "I looked and the bin is empty" is a correct observation and refusing it forces a correct agent to invent a detection. The reasoning is sound; it is also an argument *against* the specification rather than an implementation of it. §9.5 is normative and explicit — its StatusCode table lists `Bad_InvalidArgument` for "`Detections` empty; or `SubmitCorrection` supplies both or neither corrected array". Worse, **none of §9.5's argument rules were enforced anywhere** — the dispatcher forwarded every submission straight to the feedback sink, so conformance depended on whichever sink a host installed. They are Server obligations, so they now live in the dispatcher, ahead of the sink, proven by strict mocks that would throw if the sink were consulted. The usability problem is real and does not go away by conforming, so it is raised where it can be fixed: [opcua-drafts#70](https://github.com/marcschier/opcua-drafts/issues/70). ## Tracking the updated drafts The Vision, Robot Intent and AI Model Management NodeSets on `opcua-drafts` `main` have moved, partly because of feedback this branch raised. All three are re-vendored byte-identically — verified by comparing git blob hashes against the upstream files rather than by eye — and the implementations follow. **Two statements that were previously inexpressible now have a wire representation.** `SubmitDetections` takes `SceneIsEmpty` and `SubmitCorrection` takes `RetractAll`, so an agent can report "I examined this frame and there is nothing in it" — the terminating condition of the bin-picking task — and can retract a false positive by correcting a result down to nothing. Both are checked in both directions, because the flag is exactly what separates a deliberate empty observation from a lost payload: an empty array without the flag is refused, and the flag with an array attached is refused as two contradictory claims about one frame. Implemented in the dispatcher, the client, the MCP tools the agent drives, and the sample cell. Also absorbed: `LampType` and `LightingMode` became enums (§66), `ProfileName` became `DefaultProfileName` on the endpoint (distinct from the `ProfileName` a caller passes to `GetStreamEndpoint`, which keeps its name), AI `Invoke`/`InvokeAsync` gained `PayloadUri` under the same exactly-one rule, and AI `ListModels` gained a `ContinuationPoint` — because `MaxResults` alone puts everything past the bound permanently out of reach. **OpenUSD is on `0.9.0-alpha`**, verified on the running cell rather than by restore alone: D3D12 backend, 612×512 frame, the bin and all five parts in view and detected — the same result `0.8.0-alpha` produced. The one local workaround that survives is `VisionMethodArguments`, and its reason has changed rather than gone away: the builder materialises these Methods as Optional children through `CreateOrReplace`, which constructs the state object directly and never runs the generated factory that now carries the signatures. Checked rather than assumed — removing the declarations fails five of the eight `VisionMethodSurfaceTests`. ## Review feedback Three threads, all addressed in `0fe2803a3`: - **The AI Model Management work is now a library family, not a sample.** `src/Opc.Ua.AI` (model), `.Inference` (`IInferenceBackend` and backends), `.Server` (node manager) and `.Client` (discovery, reads, calls, artefact transfer), in the same shape as the Robotics and Vision families; the samples become `samples/AI/ModelManagementServer` and `ModelManagementClient`. The client library needed real work rather than relocation: its browsing surface was `private` inside the sample's scenario runner, so moving the file alone would have produced a library nobody could call. - **No cloud-vendor SDK is referenced anywhere.** Inference goes through `Microsoft.Extensions.AI` — an `IChatClient` is implemented by hosted services and on-device runtimes alike, which is exactly the property clause 8.1 asserts. `Azure.Identity` is gone: workload identity is not an Azure feature, every platform projects a token into a file and each SDK reads that file behind its own API, so the resolver reads it directly. The comment claiming `Microsoft.Extensions.AI` would force a `System.Text.Json` bump was stale — it pins 10.0.10, which this repository already centralises on. - **`BinPickingCell` and `BinPickingClient` moved into `samples/Robotics/`**, which is where they belonged: the cell was already linking the arm, its kinematics and both USD assets out of `IntentEnabledRobot`. ## Upstream Work that belonged elsewhere went elsewhere rather than being worked around locally: - **`openusd-dotnet` [#13](https://github.com/marcschier/openusd-dotnet/pull/13)** — merged and shipped in `0.8.0-alpha`. `SilkFrameCapture.Capture` rendered only the first capture of a session and silently returned a blank frame for every one after it. Also filed [#14](https://github.com/marcschier/openusd-dotnet/issues/14) (no camera-by-prim API) and [#15](https://github.com/marcschier/openusd-dotnet/issues/15) (skipped tests reported as passed — which fooled me for several runs). Both are fixed; this branch takes `0.8.0-alpha` and deletes two workarounds, including 143 lines of hand-rolled projection maths. - **`opcua-drafts` [#66](https://github.com/marcschier/opcua-drafts/issues/66)–[#69](https://github.com/marcschier/opcua-drafts/issues/69)** — a close read of the spec produced 22 candidate ambiguities; 12 were discarded as misreadings with clause citations and the remaining 10 filed. One is concrete: five members exist in the NodeId CSV that no clause documents. - **`opcua-drafts` [#70](https://github.com/marcschier/opcua-drafts/issues/70)** — §9.5 could not express an empty observation or a false-positive retraction. **Adopted**, with the explicit-flag resolution argued for: `SubmitDetections` gained `SceneIsEmpty`, `SubmitCorrection` gained `RetractAll`, and the corrected-array rule relaxed from *exactly one* to *at most one*. - **`opcua-drafts` [#71](https://github.com/marcschier/opcua-drafts/issues/71)** — the Vision NodeSet browse-named all 13 `InputArguments` / `OutputArguments` Properties in the Vision namespace rather than the base namespace, so a stack could not find them and every Method in the specification was uncallable. **Fixed**; this branch now tracks the corrected artefact. ## The defect underneath it all The end-to-end suite was removed earlier in this branch because nine of its fourteen tests failed on one symptom: a client could not open a pipeline's `Feedback` object or enumerate its `Results`, even though browsing the same parent listed both. Restoring it meant explaining that, and the explanation was not in Vision. `RelativePathElement` initialised **`IsInverse` to `true`**. A caller that sets a `ReferenceTypeId` and a `TargetName` — which reads as *"the child reached by this Reference"* — was therefore asking the Server for the **inverse** Reference, and got `Bad_NoMatch` with nothing in the request to suggest why. `ObjectTypeClient.ResolveChildNodeIdAsync` is such a caller, and it is what **every source-generated Optional-child accessor in the stack** is built on — the accessor whose own documentation cites `AlarmConditionTypeClient.GetShelvingStateAsync` as an example. Vision is merely where it surfaced, because Vision puts two Optional children on the path an off-Server agent must take. The sibling helper in `StateMachineTypeClientExtensions` already set `IsInverse = false` by hand, which is the shape of a trap rather than a default. The default is now forward, `ResolveChildNodeIdAsync` says so explicitly, and the existing test that had recorded `true` as *expected* now pins `false` with the reasoning. Of the forty `RelativePathElement` construction sites in the tree, that accessor was **the only one relying on the default at all** — every deliberate caller already set it. Two further defects were found underneath it, both real and both fixed: - The generated `CreateOrReplace` helpers construct a child state object directly and leave **`TypeDefinitionId`** unset as well as `ReferenceTypeId`. A child referenced by nothing cannot be browsed; one with no type definition is a malformed Object that a client filtering by type silently skips — which is exactly what `EnumerateResultsAsync` does. Normalisation moved to `VisionNodeManager`, so it also covers the case the builder cannot see: a result published **at runtime** by an inference provider, long after the address space was built. That path produced results a client could list but not read. - `Opc.Ua.AI` shipped without an `AssemblyInfo`, so the solution built with six `CA1014` warnings. I had initially "fixed" this in the Vision client with Browse-based fallbacks, on the strength of a probe that appeared to show `TranslateBrowsePathsToNodeIds` failing even for `Server → ServerStatus`. That probe was wrong: it used an unqualified `ReferenceTypeIds`, which inside a `Opc.Ua.Vision.*` namespace binds to *Vision's* `ReferenceTypeIds` and yields a nonsense NodeId. Chasing that to the end is what turned up the real cause. **The fallbacks are gone** — they would have hidden a stack-wide bug behind a Vision-shaped workaround, and one of them matched on browse name while ignoring the namespace index. `tests/Opc.Ua.Vision.Intent.Tests` is restored and back in `UA.slnx`, passing **14/14** on its own merit rather than through a client-side workaround. A registration test now pins the pipeline's `Feedback` and `Results` children, which nothing covered before. ## Running it, and what that found The demo's premise is that a language model **looks at the bin and reasons about what it sees**. Running it end to end showed it could not, for reasons no test covered, because every one of them lived between components. **The sample did not compile.** Two `CS8604` nullable errors with `TreatWarningsAsErrors`, so neither the cell server nor the demo could be started at all. Worth recording that my first explanation was wrong: CI *does* build this project - the Linux leg's `net10.0` pass compiles it, log-confirmed - so this is not a coverage gap in the workflow, and no workflow change was made. **`Pick` and `Place` never moved the arm.** They were a delay plus a gripper action, so the operations reported `Succeeded` while nothing travelled. They now solve IK once and interpolate in joint space. **And the parts never moved**: the arm swung but the bin stayed full, because nothing tied the held object to the world state. The connector now creates 11 monitored items where it created 6, and a scripted run delivers `SeqNo=151` across 10 messages where it delivered `SeqNo=3` across 1. **The frame was smuggled through a string field.** The media provider put a base64 *data URI* of the PNG into `VisionImageReferenceDataType.Uri` - a field meant to be a *reference*, while the bytes were already returned in the inline `ByteString`. Every response therefore shipped the image twice, and a 1,311,400-byte PNG became a 1,748,536-character string against a `MaxStringLength` of 65,535: **26.7x over**. That is why raising `MaxByteStringLength`, `MaxArrayLength` and `MaxMessageSize` never helped - none of them govern a `String`. Separately the MCP tool assigned raw bytes where `ImageContentBlock` requires base64, so the wire carried `"data":"�PNG..."` - every non-UTF-8 byte replaced by U+FFFD, destroying the image and inflating the response to 6.8 MB. **The camera was aimed at nothing.** The arm's home pose, the flange scan pose the Vision model declared and the joint angles in the USD stage were three independent claims about where the camera was, and they disagreed - the declared flange orientation pointed the view 86 degrees away from the projection camera. Solving them as one thing exposed two further constraints that are easy to miss: the camera has to sit off the tool axis or it photographs its own gripper, and the IK branch has to be elbow-back or a link parks under the camera and fills the frame with the arm's own upper arm. A third was a regression this branch introduced and `Opc.Ua.Robotics.Tests` caught - aiming a straight-down camera from a point on the base's own X-Z plane puts the wrist on the J4/J6 singularity, so every motion away from home fails to solve. A 15-degree camera roll, free because the camera still looks straight down, gets 25 degrees clear, which is the margin the poses before this branch had. **The frame and the detections were different images.** The sensor declared 2448x2048, the clip endpoint 1280x1024, and the renderer produced 640x512 - not even the same aspect ratio. The ground-truth detector projects through the declared intrinsics, so its boxes lived in 2448x2048 space while an agent was handed a 640x512 picture: *"pick the red cube you can see"* arrived with coordinates that pointed off the image. Everything now agrees at 612x512, an exact 4x4 bin of the native sensor with the intrinsics scaled to match; the native size stays in the model and serial number, where it describes the hardware rather than the image. **`LatestClip` was created and never written**, so it reported `Bad_NoDataAvailable` for the life of the Server and a consumer following the model - read the published frame, call the method only if there is none - never got a frame. The dispatcher now publishes a clip it has just encoded, for every Vision server rather than this sample alone. One genuine API gap turned up on the way: `OpcUaServerOptions` exposes `MaxByteStringLength`, `MaxArrayLength` and `MaxMessageSize` and **no `MaxStringLength`**, so a Server that legitimately needs a long string could not raise the limit through the hosting API at all. Added, with tests. It is not the fix for anything above - that was a design bug - but the missing knob was real. ## Not finished - **Two projects sit below the repository's 80% line coverage.** `Opc.Ua.Vision.OpenUsd` is at 71.1% — what is left only executes with a real OpenUSD stage and plugin tree, which is not on every build agent. `Opc.Ua.AI.Inference` is now at **86.7%**, up from 12.9%: the code this branch *wrote* is covered by 25 tests, and `RestChatCompletionsBackend` — previously the whole of the shortfall — is covered by 18 more against a stubbed `HttpMessageHandler`, exercising success, refusal, throttling, retry-after, timeout and cancellation without a network. Everything else clears the bar — `Opc.Ua.Vision.Server` 92.1%, `Opc.Ua.AI` 91.4%, `.Vision.Client` 88.1%, `Opc.Ua.Vision` 85.7%, `Opc.Ua.AI.Server` 80.1%, all measured with coverlet. The CI coverage gate reports itself as advisory and non-blocking. - **`SamplesCollected` is not counted here, and `docs/Vision.md` now says so.** Section 9.4 requires a Server to count a negative example (`SceneIsEmpty` / `RetractAll` carrying a `GroundTruthLabel`) exactly as it counts one carrying geometry. `VisionNodeManager` does not, and structurally cannot: `SamplesCollected` is a property of `LearningJobType`, which the *AI Model Management* companion defines, and Vision reaches the job through a `NodeId` value rather than a Reference **precisely so this model takes no dependency on the model that defines it**. What the Server does guarantee is that the negative example survives the hop intact — both flags are carried verbatim to `IVisionFeedbackSink` — so a host that binds a learning job has everything it needs to satisfy the clause on the counter it owns. Stated in the limitations rather than left implied. ## Related Issues Was stacked on #4195, now merged. No tracking issue yet for the Vision implementation itself — happy to open one if you would like the design recorded as an ADR before this goes further. ## Checklist - [x] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. — *Types 8562, Server 4072, Client 2123, InformationModel 1063, Robotics 540, Tools 466, Vision 442, AI 74 and the restored Vision+Intent end-to-end suite 14/14, all green on net10.0. Only `.OpenUsd` (71.1%) is below the 80% bar — see Not finished.* - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. — *`dotnet build UA.slnx -c Release`: 0 errors, 0 warnings.* - [x] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. — *net48: Types 8555, Client 2126, Vision 385 green. Two net48 Server tests fail (`CreateUserManagementSeamBindsTheModelToTheServerAsync`, and `DurableDataValueQueueVerifyReferenceBatchingAsync` only under parallel load); both were confirmed **pre-existing** by re-running them with the `IsInverse` change reverted, which fails identically.* - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. — *The net48 build legs failed because the restored end-to-end project used `ValueTask.FromResult` / `string.Create`, which do not exist on .NET Framework; fixed in `fb57522da` and verified building on every TFM. `DisposeDrainsHeldConnectionCallbackBeforeListenerDisposal` failed once on the Ubuntu Client leg — a 13 ms timing-sensitive reverse-connect disposal test that passes locally 2123/2123 and touches no browse paths.* - [ ] I have addressed **all** PR feedback received. ## Typed Vision-guided Robotics MCP workflows The branch now also makes the demo workflow first-class in the Full MCP profile: - replaces stringified Robotics intent and mission inputs with typed request schemas and exact name-or-NodeId selectors; - adds bounded operation/mission paging, stable step-to-operation identity, retained mission discovery, and `robotics_wait_mission`; - makes `vision_run_inference` return bounded typed detection, inspection, or segmentation summaries in one call; - adds same-session `robotics_vision_pick` to select a detection and submit Pick or Pick/Place work without implicitly taking authority, retrying, cancelling, or waiting; and - starts Publish workers for classic subscriptions so an independently launched OpenUSD viewer receives live robot and part updates. The migration guide documents the intentionally breaking MCP request-schema changes. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace Copilot-Session: bcbc8ac9-5abd-42a8-8bac-e10533790e2e
940 lines
39 KiB
C#
940 lines
39 KiB
C#
/* ========================================================================
|
|
* Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved.
|
|
*
|
|
* OPC Foundation MIT License 1.00
|
|
*
|
|
* Permission is hereby granted, free of charge, to any person
|
|
* obtaining a copy of this software and associated documentation
|
|
* files (the "Software"), to deal in the Software without
|
|
* restriction, including without limitation the rights to use,
|
|
* copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
* copies of the Software, and to permit persons to whom the
|
|
* Software is furnished to do so, subject to the following
|
|
* conditions:
|
|
*
|
|
* The above copyright notice and this permission notice shall be
|
|
* included in all copies or substantial portions of the Software.
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
|
|
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
|
|
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
|
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
|
* OTHER DEALINGS IN THE SOFTWARE.
|
|
*
|
|
* The complete license agreement can be found here:
|
|
* http://opcfoundation.org/License/MIT/1.00/
|
|
* ======================================================================*/
|
|
|
|
using System;
|
|
using System.Collections.Generic;
|
|
using System.Globalization;
|
|
using System.IO;
|
|
using System.Linq;
|
|
using System.Runtime.InteropServices;
|
|
using System.Text;
|
|
using System.Threading;
|
|
using System.Threading.Tasks;
|
|
#if INTENT_VIEWER_MCP
|
|
using Microsoft.AspNetCore.Builder;
|
|
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Hosting;
|
|
#endif
|
|
using Microsoft.Extensions.Logging;
|
|
#if INTENT_VIEWER_MCP
|
|
using Microsoft.Extensions.Logging.Console;
|
|
using ModelContextProtocol.Server;
|
|
#endif
|
|
using Opc.Ua;
|
|
using Opc.Ua.Client;
|
|
#if INTENT_VIEWER_MCP
|
|
using Opc.Ua.Mcp;
|
|
#endif
|
|
using Opc.Ua.OpenUsd.Client;
|
|
using Opc.Ua.Robotics.Client;
|
|
using Opc.Ua.Robotics.Client.Intent;
|
|
using Opc.Ua.RobotIntent;
|
|
|
|
namespace IntentViewerClient
|
|
{
|
|
internal sealed record IntentViewerOptions
|
|
{
|
|
public string ServerUrl { get; init; } = "opc.tcp://localhost:62840/IntentEnabledRobot";
|
|
|
|
public bool Insecure { get; init; }
|
|
|
|
public bool View { get; init; }
|
|
|
|
public string? Renderer { get; init; }
|
|
|
|
public string? FetchAssetsDirectory { get; init; }
|
|
|
|
public int Seconds { get; init; }
|
|
|
|
public UsdViewPickMode PickMode { get; init; } = UsdViewPickMode.Auto;
|
|
|
|
public bool Mission { get; init; }
|
|
|
|
public bool Mcp { get; init; }
|
|
|
|
public string CommandPrimPath { get; init; } = "/World/IntentCommand";
|
|
|
|
public string? Transport { get; init; }
|
|
|
|
public int Port { get; init; } = 5100;
|
|
|
|
public static IntentViewerOptions Parse(string[] args)
|
|
{
|
|
return new IntentViewerOptions
|
|
{
|
|
ServerUrl = GetOption(args, "--server") ?? "opc.tcp://localhost:62840/IntentEnabledRobot",
|
|
Insecure = HasFlag(args, "--insecure"),
|
|
View = HasFlag(args, "--view"),
|
|
Renderer = GetOption(args, "--renderer"),
|
|
FetchAssetsDirectory = GetOption(args, "--fetch-assets"),
|
|
Seconds = int.TryParse(
|
|
GetOption(args, "--seconds"), NumberStyles.Integer, CultureInfo.InvariantCulture, out int seconds)
|
|
? seconds
|
|
: 0,
|
|
PickMode = TryParsePickMode(GetOption(args, "--pick-mode"), out UsdViewPickMode pickMode)
|
|
? pickMode
|
|
: UsdViewPickMode.Auto,
|
|
Mission = HasFlag(args, "--mission"),
|
|
Mcp = HasFlag(args, "--mcp"),
|
|
CommandPrimPath = GetOption(args, "--command-prim") ?? "/World/IntentCommand",
|
|
Transport = GetOption(args, "--transport"),
|
|
Port = int.TryParse(
|
|
GetOption(args, "--port"), NumberStyles.Integer, CultureInfo.InvariantCulture, out int port)
|
|
? port
|
|
: 5100
|
|
};
|
|
}
|
|
|
|
private static string? GetOption(string[] args, string name)
|
|
{
|
|
for (int ii = 0; ii < args.Length - 1; ii++)
|
|
{
|
|
if (string.Equals(args[ii], name, StringComparison.OrdinalIgnoreCase))
|
|
{
|
|
return args[ii + 1];
|
|
}
|
|
}
|
|
return null;
|
|
}
|
|
|
|
private static bool HasFlag(string[] args, string name)
|
|
{
|
|
return args.Any(a => string.Equals(a, name, StringComparison.OrdinalIgnoreCase));
|
|
}
|
|
|
|
internal static bool TryParsePickMode(string? value, out UsdViewPickMode pickMode)
|
|
{
|
|
return Enum.TryParse(value, ignoreCase: true, out pickMode) &&
|
|
pickMode is UsdViewPickMode.Auto or UsdViewPickMode.Renderer or UsdViewPickMode.CommandPrim;
|
|
}
|
|
|
|
internal IntentViewerMcpTransportSelection SelectMcpTransport()
|
|
{
|
|
if (Transport is not null &&
|
|
TryParseMcpTransport(Transport, out IntentViewerMcpTransport requestedTransport))
|
|
{
|
|
if (View && requestedTransport == IntentViewerMcpTransport.Stdio)
|
|
{
|
|
return new IntentViewerMcpTransportSelection(
|
|
requestedTransport,
|
|
true,
|
|
"WARNING: --transport stdio was explicitly requested with --view. " +
|
|
"MCP stdio uses stdout for protocol frames and the in-process viewer may share that stream; " +
|
|
"protocol corruption is possible.");
|
|
}
|
|
|
|
return new IntentViewerMcpTransportSelection(
|
|
requestedTransport,
|
|
true,
|
|
$"Using explicitly requested MCP transport '{requestedTransport.ToOptionValue()}'.");
|
|
}
|
|
|
|
if (View)
|
|
{
|
|
return new IntentViewerMcpTransportSelection(
|
|
IntentViewerMcpTransport.Http,
|
|
false,
|
|
"Using MCP transport 'http' because --view is enabled. MCP stdio frames use stdout, " +
|
|
"which cannot safely coexist with the in-process OpenUSD viewer.");
|
|
}
|
|
|
|
return new IntentViewerMcpTransportSelection(
|
|
IntentViewerMcpTransport.Stdio,
|
|
false,
|
|
"Using default MCP transport 'stdio'.");
|
|
}
|
|
|
|
internal static bool TryParseMcpTransport(string? value, out IntentViewerMcpTransport transport)
|
|
{
|
|
if (string.Equals(value, "stdio", StringComparison.OrdinalIgnoreCase))
|
|
{
|
|
transport = IntentViewerMcpTransport.Stdio;
|
|
return true;
|
|
}
|
|
|
|
if (string.Equals(value, "http", StringComparison.OrdinalIgnoreCase) ||
|
|
string.Equals(value, "sse", StringComparison.OrdinalIgnoreCase))
|
|
{
|
|
transport = IntentViewerMcpTransport.Http;
|
|
return true;
|
|
}
|
|
|
|
transport = IntentViewerMcpTransport.Stdio;
|
|
return false;
|
|
}
|
|
}
|
|
|
|
internal sealed record TargetLocation(string PrimPath, NodeId LocationNodeId, string Name);
|
|
|
|
internal enum IntentViewerMcpTransport
|
|
{
|
|
Stdio,
|
|
|
|
Http
|
|
}
|
|
|
|
internal sealed record IntentViewerMcpTransportSelection(
|
|
IntentViewerMcpTransport Transport,
|
|
bool Explicit,
|
|
string Message);
|
|
|
|
internal static class IntentViewerMcpTransportExtensions
|
|
{
|
|
public static string ToOptionValue(this IntentViewerMcpTransport transport)
|
|
{
|
|
return transport switch
|
|
{
|
|
IntentViewerMcpTransport.Stdio => "stdio",
|
|
IntentViewerMcpTransport.Http => "http",
|
|
_ => throw new ArgumentOutOfRangeException(nameof(transport), transport, "Unknown MCP transport.")
|
|
};
|
|
}
|
|
}
|
|
|
|
internal static partial class Program
|
|
{
|
|
[STAThread]
|
|
public static async Task<int> Main(string[] args)
|
|
{
|
|
var options = IntentViewerOptions.Parse(args);
|
|
if (options.Transport is not null &&
|
|
!IntentViewerOptions.TryParseMcpTransport(options.Transport, out _))
|
|
{
|
|
Console.Error.WriteLine(
|
|
$"Unknown MCP transport '{options.Transport}'. Valid transports: stdio, http, sse.");
|
|
return 2;
|
|
}
|
|
|
|
IntentViewerMcpTransportSelection mcpTransport = options.SelectMcpTransport();
|
|
if (options.Mcp)
|
|
{
|
|
#if INTENT_VIEWER_MCP
|
|
Console.Error.WriteLine(mcpTransport.Message);
|
|
#else
|
|
Console.Error.WriteLine(
|
|
"MCP hosting is unavailable for this target framework. Run the sample without --mcp, " +
|
|
"or use the net8.0, net9.0, or net10.0 target framework for MCP hosting.");
|
|
return 2;
|
|
#endif
|
|
}
|
|
|
|
using ILoggerFactory loggerFactory = LoggerFactory.Create(builder =>
|
|
builder.SetMinimumLevel(LogLevel.Information));
|
|
ILogger logger = loggerFactory.CreateLogger("IntentViewerClient");
|
|
ITelemetryContext telemetry = DefaultTelemetry.Create(builder => builder.SetMinimumLevel(LogLevel.Warning));
|
|
using CancellationTokenSource lifetime = options.Seconds > 0
|
|
? new CancellationTokenSource(TimeSpan.FromSeconds(options.Seconds))
|
|
: new CancellationTokenSource();
|
|
|
|
SampleSession sample = await SampleSession.ConnectAsync(options, telemetry, lifetime.Token)
|
|
.ConfigureAwait(false);
|
|
await using (sample.ConfigureAwait(false))
|
|
{
|
|
LogConnected(logger, options.ServerUrl);
|
|
|
|
RobotIntentClient intentClient = sample.Session.RobotIntent(telemetry, sample.Streaming);
|
|
ArrayOf<RobotIntentNodeLookupEntry> controllers =
|
|
await intentClient.DiscoverControllersAsync(lifetime.Token).ConfigureAwait(false);
|
|
if (controllers.Count == 0)
|
|
{
|
|
Console.Error.WriteLine(
|
|
"No Robot Intent controllers were advertised at the conformant " +
|
|
"Server/RobotIntent/Controllers path.");
|
|
return 2;
|
|
}
|
|
|
|
RobotIntentControllerClient controller = intentClient.Controller(controllers[0].NodeId);
|
|
RobotIntentControllerInfo controllerInfo = await controller
|
|
.ReadAsync(lifetime.Token).ConfigureAwait(false);
|
|
PrintCapabilities(controllerInfo);
|
|
|
|
bool commandAllowed = false;
|
|
CommandAuthorityLease? authority = null;
|
|
#if INTENT_VIEWER_MCP
|
|
IHost? mcpHost = null;
|
|
#endif
|
|
try
|
|
{
|
|
authority = await controller.RequestAuthorityAsync(lifetime.Token).ConfigureAwait(false);
|
|
if (authority.Granted)
|
|
{
|
|
Console.Error.WriteLine("Command authority: granted for this session.");
|
|
commandAllowed = true;
|
|
}
|
|
else
|
|
{
|
|
Console.Error.WriteLine(
|
|
$"Command authority: held by {authority.CurrentOwner}; submissions may be refused.");
|
|
}
|
|
}
|
|
catch (ServiceResultException exception) when (exception.StatusCode == StatusCodes.BadUserAccessDenied)
|
|
{
|
|
Console.Error.WriteLine(
|
|
"Command authority request was denied: the connecting identity lacks the " +
|
|
"Operator role required by command Methods. Continuing in read-only mode so " +
|
|
"discovery, capabilities, facets and target mappings remain visible.");
|
|
}
|
|
|
|
try
|
|
{
|
|
#if INTENT_VIEWER_MCP
|
|
if (options.Mcp)
|
|
{
|
|
mcpHost = await StartMcpHostAsync(
|
|
mcpTransport, options, sample.Session, lifetime.Token).ConfigureAwait(false);
|
|
}
|
|
#endif
|
|
|
|
var connector = new OpenUsdConnector(sample.Session, new MockUsdSink(), enableCommands: false);
|
|
await using (connector.ConfigureAwait(false))
|
|
{
|
|
IReadOnlyList<TargetLocation> targets = await DiscoverTargetLocationsAsync(
|
|
connector, controllerInfo, lifetime.Token).ConfigureAwait(false);
|
|
if (targets.Count == 0)
|
|
{
|
|
Console.Error.WriteLine(
|
|
"No target prim to LocationType mapping was published by OpenUSD bindings.");
|
|
return 3;
|
|
}
|
|
|
|
if (options.Mission)
|
|
{
|
|
if (commandAllowed)
|
|
{
|
|
await SubmitSmallMissionAsync(controller, targets, sample.Session, lifetime.Token)
|
|
.ConfigureAwait(false);
|
|
}
|
|
else
|
|
{
|
|
Console.Error.WriteLine("Mission demo skipped because command authority was not granted.");
|
|
}
|
|
}
|
|
|
|
string outPath = PrepareLiveLayerPath(options);
|
|
if (!string.IsNullOrEmpty(options.FetchAssetsDirectory))
|
|
{
|
|
await FetchAssetsAsync(
|
|
sample.Session, options.FetchAssetsDirectory!, lifetime.Token).ConfigureAwait(false);
|
|
}
|
|
|
|
targets = await FilterTargetsWithReadablePoseAsync(sample.Session, targets, lifetime.Token)
|
|
.ConfigureAwait(false);
|
|
if (targets.Count == 0)
|
|
{
|
|
Console.Error.WriteLine(
|
|
"All published target locations had Pose values that could not be decoded; " +
|
|
"no commandable targets remain.");
|
|
return 4;
|
|
}
|
|
|
|
using var processor = new PickProcessor(
|
|
controller, sample.Session, targets, logger, commandAllowed);
|
|
string unavailable = string.Empty;
|
|
if (options.View && UsdViewHostLoader.TryLoad(out IUsdViewHost? viewHost, out unavailable))
|
|
{
|
|
await RunViewportAsync(
|
|
options, sample.Session, outPath, viewHost!, processor, telemetry, lifetime.Token)
|
|
.ConfigureAwait(false);
|
|
}
|
|
else
|
|
{
|
|
if (options.View)
|
|
{
|
|
Console.Error.WriteLine(
|
|
$"Viewport unavailable; falling back to headless mode. {unavailable}");
|
|
}
|
|
if (options.Mcp && mcpTransport.Transport == IntentViewerMcpTransport.Stdio)
|
|
{
|
|
Console.Error.WriteLine(
|
|
"Headless keyboard control is disabled while MCP stdio is active because " +
|
|
"both would read from standard input. Use --transport http for the headless " +
|
|
"menu, or drive the robot through the MCP client.");
|
|
#if INTENT_VIEWER_MCP
|
|
await WaitForMcpServerAsync(mcpHost!, lifetime.Token).ConfigureAwait(false);
|
|
#endif
|
|
}
|
|
else
|
|
{
|
|
await RunHeadlessAsync(
|
|
sample.Session, outPath, targets, processor, lifetime.Token).ConfigureAwait(false);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
finally
|
|
{
|
|
#if INTENT_VIEWER_MCP
|
|
if (mcpHost is not null)
|
|
{
|
|
await StopMcpHostAsync(mcpHost).ConfigureAwait(false);
|
|
}
|
|
#endif
|
|
|
|
if (authority is not null)
|
|
{
|
|
await authority.DisposeAsync().ConfigureAwait(false);
|
|
}
|
|
}
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
internal static Task<Pose3DDataType> ReadLocationPoseForProcessorAsync(
|
|
ISession session,
|
|
NodeId locationNodeId,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
return ReadLocationPoseAsync(session, locationNodeId, cancellationToken);
|
|
}
|
|
|
|
#if INTENT_VIEWER_MCP
|
|
private static async Task<IHost> StartMcpHostAsync(
|
|
IntentViewerMcpTransportSelection transport,
|
|
IntentViewerOptions options,
|
|
ISession session,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
IHost host = transport.Transport == IntentViewerMcpTransport.Stdio
|
|
? BuildStdioMcpHost()
|
|
: BuildHttpMcpHost(options.Port);
|
|
|
|
OpcUaSessionManager sessionManager = host.Services.GetRequiredService<OpcUaSessionManager>();
|
|
await sessionManager.RegisterExistingSessionAsync(
|
|
"intent-viewer",
|
|
session,
|
|
"Anonymous",
|
|
cancellationToken).ConfigureAwait(false);
|
|
|
|
await host.StartAsync(cancellationToken).ConfigureAwait(false);
|
|
|
|
if (transport.Transport == IntentViewerMcpTransport.Http)
|
|
{
|
|
Console.Error.WriteLine(
|
|
$"MCP server is listening on http://localhost:{options.Port}/mcp with Robotics tools.");
|
|
}
|
|
else
|
|
{
|
|
Console.Error.WriteLine("MCP server is listening on stdio with Robotics tools.");
|
|
}
|
|
|
|
return host;
|
|
}
|
|
|
|
private static IHost BuildStdioMcpHost()
|
|
{
|
|
HostApplicationBuilder builder = Host.CreateApplicationBuilder();
|
|
ConfigureMcpLogging(builder.Logging, useStdioTransport: true);
|
|
ConfigureMcpServices(builder.Services);
|
|
|
|
IMcpServerBuilder mcpServerBuilder = builder.Services
|
|
.AddMcpServer()
|
|
.WithStdioServerTransport();
|
|
ConfigureRoboticsMcpTools(mcpServerBuilder);
|
|
return builder.Build();
|
|
}
|
|
|
|
private static WebApplication BuildHttpMcpHost(int port)
|
|
{
|
|
WebApplicationBuilder builder = WebApplication.CreateBuilder();
|
|
ConfigureMcpLogging(builder.Logging, useStdioTransport: false);
|
|
ConfigureMcpServices(builder.Services);
|
|
|
|
IMcpServerBuilder mcpServerBuilder = builder.Services
|
|
.AddMcpServer()
|
|
.WithHttpTransport();
|
|
ConfigureRoboticsMcpTools(mcpServerBuilder);
|
|
|
|
WebApplication app = builder.Build();
|
|
app.MapMcp("/mcp");
|
|
app.Urls.Add($"http://localhost:{port}");
|
|
return app;
|
|
}
|
|
|
|
private static void ConfigureMcpServices(IServiceCollection services)
|
|
{
|
|
services.AddOpcUaMcpCore(new OpcUaMcpOptions { ToolProfile = McpToolProfile.Robotics });
|
|
services.AddOpcUaMcpRobotics();
|
|
}
|
|
|
|
private static void ConfigureRoboticsMcpTools(IMcpServerBuilder mcpServerBuilder)
|
|
{
|
|
mcpServerBuilder
|
|
.WithOpcUaMcpFilters()
|
|
.WithOpcUaRoboticsTools(McpToolProfile.Robotics);
|
|
}
|
|
|
|
private static void ConfigureMcpLogging(ILoggingBuilder logging, bool useStdioTransport)
|
|
{
|
|
logging.ClearProviders();
|
|
logging.SetMinimumLevel(LogLevel.Information);
|
|
logging.AddSimpleConsole(options =>
|
|
{
|
|
options.UseUtcTimestamp = true;
|
|
options.TimestampFormat = "yyyy-MM-dd HH:mm:ss ";
|
|
});
|
|
logging.Services.Configure<ConsoleLoggerOptions>(o =>
|
|
o.LogToStandardErrorThreshold = useStdioTransport ? LogLevel.Trace : LogLevel.Error);
|
|
}
|
|
|
|
private static async Task WaitForMcpServerAsync(IHost host, CancellationToken cancellationToken)
|
|
{
|
|
try
|
|
{
|
|
await host.WaitForShutdownAsync(cancellationToken).ConfigureAwait(false);
|
|
}
|
|
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
|
{
|
|
}
|
|
}
|
|
|
|
private static async Task StopMcpHostAsync(IHost host)
|
|
{
|
|
try
|
|
{
|
|
await host.StopAsync(CancellationToken.None).ConfigureAwait(false);
|
|
}
|
|
finally
|
|
{
|
|
host.Dispose();
|
|
}
|
|
}
|
|
#endif
|
|
|
|
private static void PrintCapabilities(RobotIntentControllerInfo info)
|
|
{
|
|
Console.Error.WriteLine($"Controller: {info.BrowseName} ({info.NodeId})");
|
|
Console.Error.WriteLine($"Axis count: {info.AxisCount}; queue depth: {info.MaxQueueDepth}");
|
|
Console.Error.WriteLine("Facets:");
|
|
Console.Error.WriteLine($" Base: {info.Facets.Base}");
|
|
Console.Error.WriteLine($" Trajectories: {info.Facets.Trajectories}");
|
|
Console.Error.WriteLine($" Missions: {info.Facets.Missions}");
|
|
Console.Error.WriteLine($" Mission branching: {info.Facets.MissionBranching}");
|
|
Console.Error.WriteLine($" Force control: {info.Facets.ForceControl}");
|
|
Console.Error.WriteLine($" Real-time channels: {info.Facets.RealTimeChannels}");
|
|
Console.Error.WriteLine($" Buffer-mode rule conformant: {info.Facets.EveryCapabilitySupportsAborting}");
|
|
}
|
|
|
|
private static async Task<IReadOnlyList<TargetLocation>> DiscoverTargetLocationsAsync(
|
|
OpenUsdConnector connector,
|
|
RobotIntentControllerInfo controllerInfo,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
List<OpenUsdConnector.RepresentationInfo> representations = await connector
|
|
.DiscoverAllRepresentationsAsync(cancellationToken).ConfigureAwait(false);
|
|
var locationLookup = new Dictionary<NodeId, RobotIntentNodeLookupEntry>();
|
|
foreach (RobotIntentNodeLookupEntry entry in controllerInfo.Lookups.Locations)
|
|
{
|
|
locationLookup[entry.NodeId] = entry;
|
|
}
|
|
var targets = new List<TargetLocation>();
|
|
foreach (OpenUsdConnector.BindingInfo binding in representations.SelectMany(rep => rep.Bindings))
|
|
{
|
|
if (string.IsNullOrWhiteSpace(binding.PrimPath))
|
|
{
|
|
continue;
|
|
}
|
|
if (!string.Equals(binding.SourceSemanticId, "RobotIntent.Location", StringComparison.Ordinal) &&
|
|
!string.Equals(binding.PropertyName, "inputs:robotIntentLocation", StringComparison.Ordinal))
|
|
{
|
|
continue;
|
|
}
|
|
NodeId nodeId = binding.SourceNodeId;
|
|
if (nodeId.IsNull && !string.IsNullOrWhiteSpace(binding.SourceSemanticId))
|
|
{
|
|
try
|
|
{
|
|
nodeId = NodeId.Parse(binding.SourceSemanticId);
|
|
}
|
|
catch (Exception) when (binding.SourceSemanticId is not null)
|
|
{
|
|
continue;
|
|
}
|
|
}
|
|
if (nodeId.IsNull)
|
|
{
|
|
continue;
|
|
}
|
|
if (targets.Any(t => string.Equals(t.PrimPath, binding.PrimPath, StringComparison.Ordinal)))
|
|
{
|
|
continue;
|
|
}
|
|
string name = locationLookup.TryGetValue(nodeId, out RobotIntentNodeLookupEntry? entry)
|
|
? entry.Name
|
|
: binding.PrimPath!.Split('/').LastOrDefault(static s => s.Length > 0) ?? binding.PrimPath!;
|
|
targets.Add(new TargetLocation(binding.PrimPath!, nodeId, name));
|
|
}
|
|
return [.. targets.OrderBy(t => t.Name, StringComparer.Ordinal)];
|
|
}
|
|
|
|
private static async Task<IReadOnlyList<TargetLocation>> FilterTargetsWithReadablePoseAsync(
|
|
ISession session,
|
|
IReadOnlyList<TargetLocation> targets,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
var readable = new List<TargetLocation>();
|
|
foreach (TargetLocation target in targets)
|
|
{
|
|
try
|
|
{
|
|
_ = await ReadLocationPoseAsync(
|
|
session, target.LocationNodeId, cancellationToken).ConfigureAwait(false);
|
|
readable.Add(target);
|
|
}
|
|
catch (ServiceResultException exception) when (exception.StatusCode == StatusCodes.BadUnexpectedError)
|
|
{
|
|
Console.Error.WriteLine(
|
|
$"Location {target.Name} ({target.LocationNodeId}) published a Pose value this " +
|
|
"client could not decode; it will be omitted from the command menu.");
|
|
}
|
|
}
|
|
return readable;
|
|
}
|
|
|
|
private static string PrepareLiveLayerPath(IntentViewerOptions options)
|
|
{
|
|
string root = options.FetchAssetsDirectory ?? Path.Combine(GetPrivateStateRoot(), "intent-viewer");
|
|
Directory.CreateDirectory(root);
|
|
string path = Path.Combine(root, "live.usda");
|
|
if (!File.Exists(path))
|
|
{
|
|
File.WriteAllText(path, "#usda 1.0\n(\n doc = \"OPC UA -> OpenUSD live override layer\"\n)\n");
|
|
}
|
|
return path;
|
|
}
|
|
|
|
private static async Task FetchAssetsAsync(
|
|
ISession session,
|
|
string cacheDir,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
Directory.CreateDirectory(cacheDir);
|
|
var fetcher = new OpenUsdConnector(session, new MockUsdSink(), enableCommands: false);
|
|
await using (fetcher.ConfigureAwait(false))
|
|
{
|
|
List<OpenUsdConnector.FetchedAsset> fetched =
|
|
await fetcher.FetchServedAssetsAsync(cacheDir, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
if (fetched.Count == 0)
|
|
{
|
|
Console.Error.WriteLine("Server did not advertise served OpenUSD assets.");
|
|
return;
|
|
}
|
|
WriteStageUsda(cacheDir, fetched);
|
|
Console.Error.WriteLine($"Fetched {fetched.Count} OpenUSD asset(s) into {cacheDir}.");
|
|
}
|
|
}
|
|
|
|
private static async Task RunViewportAsync(
|
|
IntentViewerOptions options,
|
|
ISession session,
|
|
string liveLayerPath,
|
|
IUsdViewHost viewHost,
|
|
PickProcessor processor,
|
|
ITelemetryContext telemetry,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
string cacheDir = options.FetchAssetsDirectory
|
|
?? Path.GetDirectoryName(liveLayerPath)
|
|
?? AppContext.BaseDirectory;
|
|
|
|
// The viewport needs the served geometry: without it only the live override layer
|
|
// composes, which carries transforms but no geometry and renders as an empty scene.
|
|
await FetchAssetsAsync(session, cacheDir, cancellationToken).ConfigureAwait(false);
|
|
|
|
string stagePath = Path.Combine(cacheDir, "stage.usda");
|
|
if (!File.Exists(stagePath))
|
|
{
|
|
Console.Error.WriteLine("No fetched stage.usda exists; viewport will open the live override layer only.");
|
|
stagePath = liveLayerPath;
|
|
}
|
|
|
|
var viewOptions = new UsdViewOptions
|
|
{
|
|
StagePath = stagePath,
|
|
Renderer = options.Renderer,
|
|
CameraPath = "/World/Camera",
|
|
Title = "OPC UA Robot Intent Viewer",
|
|
Telemetry = telemetry,
|
|
PickMode = options.PickMode,
|
|
CommandPrimPath = options.CommandPrimPath,
|
|
PrimPicked = processor.ProcessPickAsync
|
|
};
|
|
Console.Error.WriteLine("Opening OpenUSD viewport. Click a target puck to submit a Robot Intent.");
|
|
await RunViewportOnStaThreadAsync(viewHost, viewOptions, session, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
}
|
|
|
|
private static Task<bool> RunViewportOnStaThreadAsync(
|
|
IUsdViewHost viewHost,
|
|
UsdViewOptions viewOptions,
|
|
ISession session,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
var completion = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
|
|
var uiThread = new Thread(() =>
|
|
{
|
|
try
|
|
{
|
|
viewHost.RunViewport(
|
|
viewOptions,
|
|
async (sink, ct) => await StreamOpenUsdAsync(session, sink, ct).ConfigureAwait(false),
|
|
cancellationToken);
|
|
completion.TrySetResult(true);
|
|
}
|
|
#pragma warning disable CA1031 // Surfaced to the asynchronous caller.
|
|
catch (Exception exception)
|
|
#pragma warning restore CA1031
|
|
{
|
|
completion.TrySetException(exception);
|
|
}
|
|
})
|
|
{
|
|
IsBackground = false,
|
|
Name = "Intent OpenUSD viewport"
|
|
};
|
|
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
|
|
{
|
|
uiThread.SetApartmentState(ApartmentState.STA);
|
|
}
|
|
uiThread.Start();
|
|
return completion.Task;
|
|
}
|
|
|
|
private static async Task RunHeadlessAsync(
|
|
ISession session,
|
|
string liveLayerPath,
|
|
IReadOnlyList<TargetLocation> targets,
|
|
PickProcessor processor,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
_ = session;
|
|
_ = liveLayerPath;
|
|
Console.Error.WriteLine("Headless target menu:");
|
|
for (int ii = 0; ii < targets.Count; ii++)
|
|
{
|
|
Console.Error.WriteLine($" {ii + 1}. {targets[ii].Name} ({targets[ii].PrimPath})");
|
|
}
|
|
bool processed = false;
|
|
while (!cancellationToken.IsCancellationRequested)
|
|
{
|
|
int? index = await ReadTargetIndexAsync(targets.Count, processed, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
if (!index.HasValue)
|
|
{
|
|
return;
|
|
}
|
|
processed = true;
|
|
await processor.ProcessPickAsync(targets[index.Value].PrimPath, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
}
|
|
}
|
|
|
|
private static async Task<int?> ReadTargetIndexAsync(
|
|
int count,
|
|
bool allowExit,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
Console.Error.Write(allowExit ? "Choose target number, or press Enter to exit: " : "Choose target number: ");
|
|
string? line = await Task.Run(Console.In.ReadLine, cancellationToken).ConfigureAwait(false);
|
|
if (line is null || (allowExit && string.IsNullOrWhiteSpace(line)))
|
|
{
|
|
return null;
|
|
}
|
|
if (int.TryParse(line, NumberStyles.Integer, CultureInfo.InvariantCulture, out int selected) &&
|
|
selected >= 1 &&
|
|
selected <= count)
|
|
{
|
|
return selected - 1;
|
|
}
|
|
if (allowExit)
|
|
{
|
|
Console.Error.WriteLine("No valid selection supplied; exiting headless mode.");
|
|
return null;
|
|
}
|
|
Console.Error.WriteLine("No valid selection supplied; using the first published target for headless automation.");
|
|
return 0;
|
|
}
|
|
|
|
private static async Task StreamOpenUsdAsync(
|
|
ISession session,
|
|
IUsdSink sink,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
var connector = new OpenUsdConnector(session, sink, enableCommands: false);
|
|
await using (connector.ConfigureAwait(false))
|
|
{
|
|
await StreamConnectorUntilCancelledAsync(
|
|
connector.StartAsync,
|
|
connector.StopAsync,
|
|
cancellationToken).ConfigureAwait(false);
|
|
}
|
|
}
|
|
|
|
internal static async Task StreamConnectorUntilCancelledAsync(
|
|
Func<CancellationToken, Task> startAsync,
|
|
Func<CancellationToken, Task> stopAsync,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
await startAsync(cancellationToken).ConfigureAwait(false);
|
|
try
|
|
{
|
|
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken).ConfigureAwait(false);
|
|
}
|
|
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
|
{
|
|
// Viewport shutdown is the expected end of the live stream.
|
|
}
|
|
finally
|
|
{
|
|
await stopAsync(CancellationToken.None).ConfigureAwait(false);
|
|
}
|
|
}
|
|
|
|
private static async Task SubmitSmallMissionAsync(
|
|
RobotIntentControllerClient controller,
|
|
IReadOnlyList<TargetLocation> targets,
|
|
ISession session,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
if (targets.Count < 2)
|
|
{
|
|
Console.Error.WriteLine("Mission demo skipped: fewer than two target locations were published.");
|
|
return;
|
|
}
|
|
Pose3DDataType first = await ReadLocationPoseAsync(session, targets[0].LocationNodeId, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
Pose3DDataType second = await ReadLocationPoseAsync(session, targets[1].LocationNodeId, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
MissionDataType mission = RobotIntentBuilder.Mission("viewer-mission")
|
|
.ReleasedStep("base", RobotIntentBuilder.LinearMove(first, 0.25).Build())
|
|
.HorizonStep("horizon", RobotIntentBuilder.LinearMove(second, 0.25).Build())
|
|
.Build();
|
|
MissionSubmissionResult result = await controller.SubmitMissionAsync(mission, cancellationToken)
|
|
.ConfigureAwait(false);
|
|
Console.Error.WriteLine(result.Accepted
|
|
? $"Mission accepted: {result.MissionId} operation {result.Operation}."
|
|
: $"Mission refused: {result.Failure} {result.Message.Text}");
|
|
}
|
|
|
|
private static async Task<Pose3DDataType> ReadLocationPoseAsync(
|
|
ISession session,
|
|
NodeId locationNodeId,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
NodeId poseNodeId = await BrowseChildAsync(
|
|
session, locationNodeId, "Pose", cancellationToken).ConfigureAwait(false);
|
|
DataValue value = await session.ReadValueAsync(poseNodeId, cancellationToken).ConfigureAwait(false);
|
|
Pose3DDataType pose = null!;
|
|
if (value.WrappedValue.TryGetValue(out pose!, session.MessageContext) && pose is not null)
|
|
{
|
|
return pose;
|
|
}
|
|
throw ServiceResultException.Create(
|
|
StatusCodes.BadUnexpectedError, "Location Pose did not contain Pose3DDataType.");
|
|
}
|
|
|
|
private static async ValueTask<NodeId> BrowseChildAsync(
|
|
ISession session,
|
|
NodeId parent,
|
|
string browseName,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
ArrayOf<ReferenceDescription> references =
|
|
await BrowseReferencesAsync(session, parent, cancellationToken).ConfigureAwait(false);
|
|
foreach (ReferenceDescription reference in references)
|
|
{
|
|
if (string.Equals(reference.BrowseName.Name, browseName, StringComparison.Ordinal))
|
|
{
|
|
return ExpandedNodeId.ToNodeId(reference.NodeId, session.NamespaceUris);
|
|
}
|
|
}
|
|
throw ServiceResultException.Create(
|
|
StatusCodes.BadNodeIdUnknown, "Child {0} was not found on {1}.", browseName, parent);
|
|
}
|
|
|
|
private static async Task<ArrayOf<ReferenceDescription>> BrowseReferencesAsync(
|
|
ISession session,
|
|
NodeId parent,
|
|
CancellationToken cancellationToken)
|
|
{
|
|
(ArrayOf<ArrayOf<ReferenceDescription>> results, ArrayOf<ServiceResult> _) = await session
|
|
.ManagedBrowseAsync(
|
|
null,
|
|
null,
|
|
[parent],
|
|
0,
|
|
BrowseDirection.Forward,
|
|
global::Opc.Ua.ReferenceTypeIds.HierarchicalReferences,
|
|
includeSubtypes: true,
|
|
0,
|
|
cancellationToken).ConfigureAwait(false);
|
|
if (results.Count > 0)
|
|
{
|
|
return results[0];
|
|
}
|
|
return [];
|
|
}
|
|
|
|
private static void WriteStageUsda(string cacheDir, List<OpenUsdConnector.FetchedAsset> fetched)
|
|
{
|
|
OpenUsdConnector.FetchedAsset? root = fetched.Find(asset => asset.Kind == OpenUsdAssetKind.RootLayer);
|
|
string rootName = root != null ? Path.GetFileName(root.LocalPath) : "base.usda";
|
|
var builder = new StringBuilder();
|
|
builder.Append("#usda 1.0\n(\n")
|
|
.Append(
|
|
" doc = \"Self-contained OpenUSD stage: server-delivered base layers + live override.\"\n")
|
|
.Append(" subLayers = [\n @./live.usda@,\n @./")
|
|
.Append(rootName).Append("@\n ]\n")
|
|
.Append(")\n");
|
|
File.WriteAllText(Path.Combine(cacheDir, "stage.usda"), builder.ToString());
|
|
string livePath = Path.Combine(cacheDir, "live.usda");
|
|
if (!File.Exists(livePath))
|
|
{
|
|
File.WriteAllText(livePath, "#usda 1.0\n(\n doc = \"OPC UA -> OpenUSD live bindings\"\n)\n");
|
|
}
|
|
}
|
|
|
|
private static string GetPrivateStateRoot()
|
|
{
|
|
string baseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);
|
|
if (string.IsNullOrEmpty(baseDirectory))
|
|
{
|
|
baseDirectory = AppContext.BaseDirectory;
|
|
}
|
|
string root = Path.Combine(baseDirectory, "OPC Foundation", "IntentViewerClient");
|
|
Directory.CreateDirectory(root);
|
|
return root;
|
|
}
|
|
|
|
[LoggerMessage(EventId = IntentViewerClientEventIds.Connected, Level = LogLevel.Information,
|
|
Message = "Connected to {ServerUrl}.")]
|
|
private static partial void LogConnected(ILogger logger, string serverUrl);
|
|
}
|
|
}
|