Generate deployables and Universal Destination Writers

Generate deployables turns the current catalog and/or saved Output Target definitions into a portable runner. The runner can keep CSV/Excel audit outputs and can also route one or more logical target definitions into one physical destination.

DSE now uses a Universal Destination Writer Framework. Every Output Target preset in the current catalog has an executable writer path. The implementation is technology-aware: native managed providers are used where practical, ODBC is used for vendor databases/warehouses that expose a standard write driver, documented HTTP/OData APIs are used for API-first systems, cloud SDKs are used for object stores, file writers are used for filesystem targets, and mainframe/custom targets use an explicit gateway or ODBC bridge.

Important: “Runtime writer” means the generated runner knows how to execute the route. It does not mean that every external prerequisite is bundled. ODBC destinations still need the appropriate vendor ODBC driver. SaaS/API destinations need a valid endpoint and authorization. VSAM/IMS/CICS require an approved site gateway or an ODBC path supported by the customer environment.

1. Recommended workflow

1. Discover/configure sources and build the Schema/Data Catalog.
2. Create/review reusable mappings in Compose targets.
3. Open Generate deployables.
4. Add one or more saved Output Target definitions.
5. Configure the proposed physical destination before activating it.
6. Review the writer technology shown beside the destination preset.
7. Edit destination object names and object types.
8. Choose Append or Replace when the selected writer supports both.
9. Enable Route selected targets to one destination only when you want the generated runner to perform the physical write.
10. Configure and test the destination connection.
11. Select target platforms/RIDs and optional CSV/Excel audit outputs.
12. Generate, run, review results, and reconcile destination counts.

2. Logical targets versus the physical destination

A saved Output Target is a logical data contract. It describes which source fields map to which logical destination fields. It does not have to be the technology that receives the data at runtime.

Example:

MongoDB + Couchbase + InfluxDB -> PostgreSQL

DSE can create a route such as:

• MongoDB logical target -> MongoDB table
• Couchbase logical target -> Couchbase table
• InfluxDB logical target -> InfluxDB table

Or:

Oracle + MongoDB + DynamoDB -> Redis

with one Redis Hash destination per selected logical target.

The route manifest keeps source-target provenance, source object identity, destination object identity, destination field names, types, nullable/key metadata, and write mode so the generated runner does not blindly merge unrelated records.

3. Configure first, activate second

The destination editor remains usable while Route selected targets to one destination is unchecked.

You can therefore:

• compare destination presets;
• inspect writer technology/capabilities;
• rename destination objects;
• change object types;
• choose a write mode;
• prepare the connection;

without activating an external write.

Checking Route selected targets to one destination activates the physical route for generation. This separation is deliberate: route design is safe and reversible; execution is explicit.

4. Destination writer technologies

The capability badge includes the writer technology.

Why DSE does not pretend every writer is identical

Different destination technologies have different physical semantics. A SQL table supports columns and transactions; Redis stores key/value structures; InfluxDB uses measurements; S3 writes objects; Salesforce exposes record APIs; CICS usually requires a site-controlled transaction/gateway. DSE uses one canonical routing model but delegates the physical operation to the appropriate adapter.

5. Destination object routing table

Each row represents one selected logical target.

Input target

Shows the saved target and mapped-field count.

Destination object

This is directly editable. The generated value is only a safe default. Click the field and enter the physical object name you want.

Names must be:

• non-empty;
• unique within the route;
• valid for the selected destination;
• stable when downstream consumers depend on them.

Object type

The object-type choices are constrained by the destination writer. Examples:

• relational/warehouse ODBC or native -> Table;
• MongoDB -> Collection;
• CosmosDB -> Container;
• DynamoDB/Cassandra -> Table;
• Couchbase/ArangoDB -> Collection;
• CouchDB -> Database;
• Redis -> Hash;
• Elasticsearch/OpenSearch -> Index;
• Neo4j -> Label;
• InfluxDB -> Measurement;
• Salesforce -> Object;
• Dynamics 365 -> Entity;
• SharePoint -> List or DocumentLibrary;
• ServiceNow -> Table;
• Workday -> BusinessObject;
• SAP OData -> EntitySet;
• S3/GCS -> Bucket or Object;
• Azure Blob -> Container or Blob;
• ADLS Gen2 -> FileSystem, Folder, or File;
• File/SMB/NFS -> Folder or File;
• Excel -> Sheet;
• IBM i -> Table or PhysicalFile;
• VSAM -> KSDS, ESDS, or RRDS through the configured gateway;
• IMS -> Database or Segment through the configured gateway;
• CICS -> Transaction through the configured gateway.

6. Canonical record model

Source connectors do not write directly into destination-specific code. The generated runner first converts selected rows/documents into a neutral DSE record envelope containing:

• stable DSE record id;
• source name/type/object;
• logical source target;
• mapped destination fields and values;
• key/provenance metadata.

The destination adapter then converts that canonical record to the physical destination representation. This is what allows a relational source, a document source and a time-series source to participate in the same destination route without creating provider-to-provider combinations for every possible pair.

7. Append and Replace

Append

Preserves existing destination data and adds the current routed records.

Replace

For destinations where DSE can define a safe whole-object replacement policy, DSE clears the route-owned object/prefix once per run and then writes the current data.

Examples:

• relational table -> delete existing rows, then insert;
• MongoDB collection -> clear records, then write;
• search index -> recreate/clear the route index;
• Redis -> remove keys in the route prefix;
• object storage -> remove route prefix/objects before writing the replacement output;
• file -> replace the route file;
• InfluxDB -> delete the route measurement over the configured bucket window before writing.

API-first SaaS safety

For Salesforce, Dynamics 365 CRM, SharePoint, ServiceNow, Workday and SAP OData, destructive full-object Replace is intentionally not exposed by the direct API writer. These systems often require business-specific delete/archive rules. Use Append, or route through an approved custom gateway that implements a reviewed replacement policy.

8. Destination connection strings

The destination connection can be embedded for a controlled lab/demo, but production deployments should normally inject it at runtime:

DSX_DESTINATION_CONNECTION_STRING

Native examples

• PostgreSQL: Host=db01;Port=5432;Database=target;Username=user;Password=...;
• SQL Server: Server=db01,1433;Database=target;User Id=user;Password=...;Encrypt=True;TrustServerCertificate=True;
• MongoDB: mongodb://user:password@host:27017/database
• Redis: Host=redis01;Port=6379;Password=...;Database=0;

ODBC

Use an explicit vendor connection string, for example:

Driver={Vendor ODBC Driver};Server=...;Database=...;Uid=...;Pwd=...;

or:

DSN=ProductionWarehouse;Uid=...;Pwd=...;

DSE intentionally does not guess vendor driver names because that would make a generated package machine-specific and unreliable.

REST/OData

Common keys understood by the universal API writer include:

• Url= / Endpoint= / BaseUrl=;
• Token= / AccessToken= / BearerToken=;
• Username= and Password=;
• ApiKey=;
• destination-specific keys such as Bucket=, Org=, Database=, Scope=, Key=, or EndpointTemplate=.

Gateway

For VSAM/IMS/CICS/custom systems, use either:

• Endpoint=https://gateway/...;Token=...; or
• a valid ODBC Driver=/DSN= connection when the site provides an ODBC write path.

The gateway receives a canonical DSE envelope containing destination target/type/object, write mode, source target and mapped records.

9. Provisioning behavior

Writers attempt only the provisioning that is safe and practical for their technology.

Examples:

• relational -> create table when absent;
• MongoDB -> collection is created by the server on first write;
• DynamoDB -> create table when absent;
• Cassandra -> create keyspace/table when absent;
• CouchDB/ArangoDB/CosmosDB/search -> create route object when the API supports it;
• object stores -> create bucket/container/filesystem where permissions allow;
• files -> create directories/files;
• SaaS record APIs -> do not create Salesforce objects, ServiceNow table definitions, etc.; those schemas must already exist or be provisioned by the platform's administrative process.

Provisioning failures are surfaced as runtime errors; DSE does not silently continue when the route destination cannot be created or accessed.

10. Test connection

Use Test connection before generation whenever possible.

DSE chooses the test method based on the writer technology:

• native connector/provider test when available;
• ODBC open for ODBC writers;
• HTTP endpoint/authentication test for REST/OData/gateway writers;
• writable-path probe for file targets;
• provider-specific tests for configured object stores where available.

A successful connectivity test confirms reachability/authentication at that moment; it does not guarantee that the account has every create/delete/write permission required by the selected route.

11. CSV and Excel audit outputs

CSV/Excel remain independent outputs. They can be generated:

• without destination routing;
• alongside a physical route for audit/review;
• with a low Max rows per object value for trial runs.

For migration work, retaining a small audit output is useful for comparing source samples with the physical destination.

12. Platform/RID selection

Typical runtime identifiers:

• win-x64
• win-arm64
• linux-x64
• linux-arm64

Remember that ODBC writers depend on the ODBC driver installed on the machine where the generated runner is executed. A driver available on the DSE design workstation is not automatically available inside another server/container.

13. AI Advisor integration

In Destinations, select exactly one AI recommendation using the radio button. The selection is staged automatically for the next Generate Deployables session; Use selected destination in Deployables can be used to confirm it explicitly. Create draft targets now creates or refreshes only the selected destination target, avoiding repeated AI-draft accumulation.

AI Advisor and Generate Deployables share the same Output Target and route model.

The Advisor can:

• analyze metadata/overlap;
• recommend a destination model/technology;
• create draft Output Targets;
• send the best route to Generate Deployables.

The AI does not silently execute a write. The user still reviews:

• destination technology;
• writer adapter type;
• destination object names/types;
• Append/Replace availability;
• connection/credentials;
• generated package;

before running it.

Because every current output preset now has a writer path, AI recommendations no longer need to be converted to an unrelated destination merely because the original technology was “Plan only.” The recommendation can remain the physical target, subject to its runtime prerequisites (ODBC driver, API token, gateway, etc.).

14. Generated runner artifacts

When destination routing is enabled, the generated schema/summary records:

• destination target system;
• writer technology;
• writer capability summary;
• write mode;
• destination database/service/path;
• per-target destination objects and mappings;
• redacted route information when credentials are excluded.

This makes generated packages easier to review and audit without exposing secrets.

15. Readiness and validation

Generation is blocked when required configuration is incomplete. Common causes include:

• no input targets/catalog selection;
• no platform/RID;
• no output folder;
• routing enabled without a destination preset;
• blank/duplicate destination object names;
• routed target has no mapped fields;
• unsupported write mode for the selected SaaS/API destination;
• missing required output mode.

Runtime validation can still fail later if an external prerequisite is unavailable, for example a missing ODBC driver, invalid token, inaccessible gateway or insufficient write permission.

16. Running and reviewing

After generation:

1. Open the generation summary.
2. Use Run now for a local execution when appropriate.
3. Use View results after output exists.
4. Review run-summary.json, CSV/XLSX audit output and route metadata.
5. Verify physical destination record counts independently.
6. For important migrations, reconcile counts/keys and retain logs before cutover.

17. Troubleshooting

Destination preset cannot be changed

The destination editor is intentionally editable while routing is off. If it is locked, install a build containing the destination-dropdown fix. The route checkbox activates execution; it should not lock configuration.

Destination object text/caret is hard to see

Use a build containing the route-editor row-height/caret fix. Destination rows use dedicated editors with increased height and direct two-way binding.

ODBC writer reports missing Driver/DSN

Install the correct vendor ODBC driver and provide an explicit ODBC connection string in the override or DSX_DESTINATION_CONNECTION_STRING.

REST/OData writer receives 401/403

Verify endpoint, token/user credentials, tenant/site context and write scopes/roles. A token that can read metadata may not have create/update permissions.

SaaS destination does not offer Replace

This is intentional. Full destructive replacement requires application-specific business rules. Use Append or an approved gateway.

VSAM/IMS/CICS asks for a gateway

These technologies normally require installation-specific transactional/host access. Configure the approved HTTP gateway or an available ODBC write path; DSE will send the canonical mapped record envelope to that bridge.

Generated runner says destination connection is required

Set DSX_DESTINATION_CONNECTION_STRING, or explicitly enable connection-string embedding for a controlled environment.

Replace left old append segments in object storage

Use the latest Universal Writer build. Replace is route-prefix scoped and removes previous route-owned segments before writing the replacement payload. Never point a route prefix at unrelated shared data.

18. Production recommendations

• Keep external writes explicit and approval-gated.
• Prefer runtime secret injection over embedded credentials.
• Test with a low row limit first.
• Use stable, unique object names.
• Start with Append unless Replace is intentional and reviewed.
• Use dedicated destination prefixes/tables/collections for migrations.
• Ensure vendor ODBC drivers exist on every deployable host/RID where needed.
• Use least-privilege API/cloud credentials.
• Keep gateway endpoints allow-listed and authenticated.
• Retain CSV/Excel/run-summary artifacts for audit.
• Reconcile record counts and critical keys before cutover.
• Treat writer adapters that depend on a vendor/cloud API as versioned integrations and regression-test them when the vendor API/driver changes.

19. Enterprise reliability, lineage and restart safety

Generated deployables can now include an enterprise reliability layer in addition to CSV/Excel and physical destination routing.

When enabled, every run receives a unique Run ID and produces structured governance artifacts under output/audit/<run-id>/:

• lineage.jsonl — one run/selection event stream containing source identity, selected fields, destination route, mapped destination objects, row counts, writer technology, write mode, status and a mapping fingerprint;
• run-summary.json — a run-scoped copy of the execution summary kept inside the audit directory so later executions cannot overwrite the historical governance record;
• run-manifest.json — a compact run-level governance summary with source/destination counts, retries, resume information and SHA-256 hashes for generated data files;
• integrity.sha256 — hashes for the run-scoped summary, lineage stream, manifest and generated audit/data files.

The lineage artifacts intentionally exclude connection strings and credentials.

Run lease and heartbeat

When Enable run lease and heartbeat is selected, the runner creates a schema-scoped lease plus heartbeat in the state directory. A second runner using the same state directory and schema cannot start until the active lease is released or becomes stale. For active/passive scheduler designs, configure the same durable DSX_STATE_DIR on both nodes.

Useful overrides:

• DSX_RUN_ID or --run-id — correlate the runner with an external scheduler/job id;
• DSX_STATE_DIR or --state-dir — durable/shared state location;
• DSX_RUN_LEASE, --lease, --no-lease;
• --lease-timeout-seconds and --heartbeat-seconds.

Retries

Transient source-read failures are retried with bounded exponential backoff. Destination writes are retried only when the writer advertises transaction or upsert semantics that make replay reasonably safe. This avoids blindly replaying non-idempotent Append calls.

Resume checkpoints

Resume completed destination selections stores successful destination-selection checkpoints. If a run fails and is restarted with resume enabled, the runner can avoid repeating destination writes already completed while still rebuilding CSV/Excel audit outputs.

Resume is intentionally disabled for Replace routes because a new process could clear a destination object before replaying a partial replacement.

Important CDC distinction

These checkpoints improve restart/replay safety, but they are not log-based Change Data Capture (CDC). The current source read mode remains Snapshot. Production CDC requires source-specific readers for transaction logs/change feeds and durable provider cursors (for example SQL Server CDC LSNs, PostgreSQL WAL/logical replication, MongoDB resume tokens, Cosmos Change Feed continuations and DynamoDB stream shard positions).

See Docs/Enterprise-Reliability-CDC-Governance-HA-Plan.md for the production CDC and distributed-orchestration roadmap.

<!-- DSE_DEPLOYABLE_CERTIFICATION_HELP_V1 -->

20. Certified execution of generated Windows and Linux deployables

Use this section after Generate deployables completes. It documents the certification/run workflow for the generated win-x64 and native linux-x64 packages. These are generated integration runners, not the DSE commercial Windows/Container installer SKUs.

20.1 Windows deployable (windows/win-x64)

Copy the generated Windows folder to the certification machine (or extract the transferred archive), then work from the windows\win-x64 directory.

Verify these generated files are present before the first run:

• DataSourceExplorer.Runner.exe
• manager.cmd
• run.cmd
• schema-catalog.json
• source-connections.json
• destination-route.json — when destination routing is enabled

Start the professional local Runtime Manager:

cd /d C:\Path\To\deployables\windows\win-x64
manager.cmd

manager.cmd launches DataSourceExplorer.Runner.exe --manager. The manager binds to 127.0.0.1 and normally opens the default browser automatically. If the browser does not open, run the executable directly and use the Local management URL printed in the console.

DataSourceExplorer.Runner.exe --manager

1. Confirm Runtime status is READY and there is no active execution.
2. In Connections, confirm each required source and destination is Configured. Leave protected fields blank when you want to keep the saved/embedded value.
3. For the first certification run, use Manual only scheduling and set the output directory to output.
4. Select Run now and wait for the status to return to READY.
5. Confirm Last result and Run history show Success, then independently verify the expected output/destination data.

For a one-time headless execution:

run.cmd

The generator first attempts a self-contained publish. If it had to fall back to a framework-dependent publish, install the matching .NET 8 and ASP.NET Core 8 runtime on the target machine.

20.2 Linux deployable (linux/linux-x64) — validated Ubuntu Hyper-V workflow

The following procedure follows the validated Ubuntu Server 26.04.1 LTS / Microsoft Hyper-V guide supplied for certification. It runs the native Linux deployable; Docker is not required for this workflow.

Preferred secure browser access from Windows

Instead of exposing the manager on the VM network, you can create an SSH local port forward from Windows PowerShell after reading the manager port from the Linux log:

ssh -L 18093:127.0.0.1:<manager-port> <linux-user>@<VM-IP>

Then open http://127.0.0.1:18093/ on Windows. Keep the SSH session open while using the manager. The socat relay described below remains useful on an isolated/trusted certification network.

Purpose

This procedure creates a lightweight Ubuntu Server virtual machine and runs the native Linux deployable produced by DSE. It mirrors the Windows deployable workflow: copy the generated package, start the DSE Deployable Runtime Manager, open the manager from the Windows host, run the deployable, and verify the result.

The package described here is a native linux-x64 deployable. Docker is not required for this workflow.

Prerequisites

• Windows host with Hyper-V enabled
• Ubuntu Server LTS ISO
• DSE-generated linux folder containing linux/linux-x64
• At least 4 vCPU, 4 GB RAM, and a 40 GB virtual disk
• Network access from the VM to the required DSE source/destination systems

Windows → Hyper-V → Ubuntu → DSE Runtime Manager → Browser

1. Create the Hyper-V virtual machine

Create a Generation 2 VM with the following baseline settings:

During Ubuntu installation, select Install OpenSSH server. After installation, remove/eject the ISO and boot from the virtual hard disk.

2. Prepare Ubuntu

Log in to Ubuntu and update the operating system. Then verify CPU, memory, disk, and IP address.

sudo apt update
sudo apt upgrade -y
hostname -I
nproc
free -h
df -h

If the 40 GB VHDX was created but the root logical volume is only about 18-19 GB, expand the LVM root volume to consume the free space.

sudo vgs
sudo lvextend -l +100%FREE -r /dev/ubuntu-vg/ubuntu-lv
df -h

3. Connect from Windows with SSH

Use the current VM IP reported by hostname -I. Hyper-V Default Switch addresses can change after a reboot.

ssh <linux-user>@<VM-IP>

4. Copy the generated Linux folder

Run ssh/scp from Windows PowerShell, not from the Ubuntu shell. DSE generates a linux folder, not a linux.zip file. Replace the example path, user, and IP with the actual values.

ssh <linux-user>@<VM-IP> "rm -rf /home/<linux-user>/dse-deploy/linux && mkdir -p /home/<linux-user>/dse-deploy"
scp -r "C:\Path\To\GeneratedDeployables\...\linux" <linux-user>@<VM-IP>:/home/<linux-user>/dse-deploy/

If Linux reports "Could not resolve hostname c:", the scp command was run from Ubuntu. Exit to Windows PowerShell and run it there.

5. Prepare the deployable

Install socat if you will use the relay method, enter the copied linux-x64 directory, and apply Linux execute permissions. Current DSE-generated shell scripts are UTF-8 without a BOM, so no extraction or BOM-removal step is required.

sudo apt update && sudo apt install -y socat
cd ~/dse-deploy/linux/linux-x64
chmod +x DataSourceExplorer.Runner manager.sh run.sh
ls -l DataSourceExplorer.Runner manager.sh run.sh

6. Verify source/destination connectivity

Before starting DSE, test that the Ubuntu VM can reach the required data source or destination port. The example below uses MongoDB on port 27017.

timeout 5 bash -c '</dev/tcp/<source-host>/<source-port>' \
  && echo 'Port reachable' \
  || echo 'Port NOT reachable'

# MongoDB example
timeout 5 bash -c '</dev/tcp/192.168.50.124/27017' \
  && echo 'MongoDB port reachable' \
  || echo 'MongoDB port NOT reachable'

7. Start the DSE Deployable Runtime Manager

Start the manager and read its console log. The manager binds to 127.0.0.1 and selects a local port.

cd ~/dse-deploy/linux/linux-x64
./manager.sh
sleep 2
cat .dse-manager/manager-console.log
curl http://127.0.0.1:<manager-port>/ | head

A successful log contains a line similar to: Local management URL: http://127.0.0.1:51827. The port can change after restart.

8. Make the manager available to the Windows browser

Use socat on Ubuntu to relay a stable VM port (18093) to the manager's current loopback port. The commands below detect both the VM IP and the manager port automatically.

cd ~/dse-deploy/linux/linux-x64
VM_IP=$(hostname -I | awk '{print $1}')
MANAGER_PORT=$(grep -oE '127\.0\.0\.1:[0-9]+' .dse-manager/manager-console.log | tail -1 | cut -d: -f2)
nohup socat TCP-LISTEN:18093,bind="$VM_IP",reuseaddr,fork TCP:127.0.0.1:"$MANAGER_PORT" > ~/dse-relay.log 2>&1 &
echo "Open: http://$VM_IP:18093/"
ss -ltnp | grep 18093

Use this relay only on a trusted test network. It makes the manager reachable on the VM network interface while socat is running.

9. Run the deployable from the UI

1. Open the URL printed by the relay command in Edge or Chrome.
2. Confirm Runtime status shows READY and no active execution.
3. In Connections, confirm the source and destination show Configured. Leave protected connection fields blank when you want to keep the embedded values.
4. Set Scheduler configuration to Manual only for the first test. Set Output directory to output and save the runtime settings.
5. Select Run now. Wait for the status to return to READY.
6. Confirm Last result and Run history show Success, then verify the expected output files and destination data.

10. Stop, restart, and clean up

The VM IP and manager port can change after a VM restart. Start manager.sh again, read the new port, and restart the socat relay. To stop the relay:

pkill -f 'socat TCP-LISTEN:18093' || true

Verification checklist

• Ubuntu boots from the VHDX and SSH login works
• Root filesystem has adequate free space
• Data source/destination port is reachable from Ubuntu
• DataSourceExplorer.Runner, manager.sh, and run.sh are executable
• manager-console.log reports a local management URL
• Windows browser opens http://<VM-IP>:18093/
• Runtime status returns to READY after Run now
• Run history shows Success and expected output/destination data is present

Troubleshooting

For product help integration, use the localized HTML page from this package. Keep commands unchanged; localize the explanatory text and UI guidance only.