Data engineering commands

These examples assume an authenticated active saved profile with the intended organization and project. Confirm it with tp context show before running commands. Add root --json before the command when machine parsing needs JSON output; use root --profile <name> only to override the active profile.

Check context and available resources

tp auth whoami
tp context show
tp datalake tree --path data/
tp datalake list --path data/
tp db tables list
tp mlops data datastores

Data Lake

tp datalake preview --path raw/source.csv
tp datalake preview --path raw/source.csv --output /tmp/source.csv
tp datalake resolve-uri --path raw/source.csv
tp datalake upload-file --local-path ./clean.csv --remote-path processed/clean.csv --dry-run
tp datalake upload-file --local-path ./clean.csv --remote-path processed/clean.csv
tp datalake upload-dir --local-dir ./outputs --remote-prefix processed/run-001 --dry-run
tp datalake move --source-path staging/a.csv --destination-path archive/a.csv --dry-run
tp datalake delete --path staging/old.csv --dry-run

Shared DB

tp db tables list
tp db tables get --table-id <table_id>
tp db tables create --body @table.json --dry-run
tp db tables create --body @table.json
tp db rows query --table-id <table_id> --body '{"filters":[],"sort":[],"page":1,"page_size":5}'
tp db rows create --table-id <table_id> --body @row.json --dry-run
tp db rows create-many --table-id <table_id> --body @rows.json --dry-run
tp db import apply --file ./import.xlsx --dry-run
tp db import apply --file ./import.xlsx

Lakehouse and Databricks

Default to tp lakehouse for Databricks/Lakehouse work. These commands call ThinkPlatform’s project-scoped Databricks APIs and work in managed Agent Workspaces without requiring direct Databricks CLI auth. Inventory and context:

tp lakehouse status
tp lakehouse catalog show
tp lakehouse schemas list
tp lakehouse tables list --schema bronze
tp lakehouse tables get bronze.example_table
tp lakehouse compute list
tp lakehouse workspace ls

SQL validation and project-scoped writes:

tp lakehouse tables get bronze.example
tp lakehouse sql preview --query "SELECT key, event_time FROM ${TP_DATABRICKS_CATALOG}.bronze.example WHERE event_date >= DATE '2026-01-01' LIMIT 100"
tp lakehouse sql preview --file ./query.sql
tp lakehouse sql preview --file ./query.sql --fresh
tp lakehouse sql execute --statement "CREATE SCHEMA IF NOT EXISTS ${TP_DATABRICKS_CATALOG}.silver" --dry-run
tp lakehouse schemas create silver --comment "Cleaned and conformed data" --dry-run

Preview uses the shared ThinkPlatform cache by default. Use --fresh only after a write when a new Databricks execution is required. In a managed ThinkPlatform Agent, call the corresponding SQL preview MCP system tool with fresh=false by default and fresh=true only for that explicit read-after-write case. Canvas dashboard handoff:

tp canvas dashboard scaffold \
  --title "Gold Metrics Dashboard" \
  --dataset-id gold_metrics \
  --query @gold_metrics.sql \
  --filter dateRange:period:event_date \
  --dry-run
tp canvas dashboard schema gold_metrics --definition @page.json
tp canvas dashboard query gold_metrics --definition @page.json --filters @filters.json --limit 100
tp canvas dashboard lint --definition @page.json

Use this handoff when a dashboard needs global filters across multiple widgets. The dataset query should include named SQL parameters that match each binding target, for example :event_date_start and :event_date_end for a dateRange binding with target event_date. Notebook and job workflows:

tp lakehouse notebooks upsert "${TP_DATABRICKS_PROJECT_ROOT}/demo" --file ./demo.py --dry-run
tp lakehouse notebooks upsert "${TP_DATABRICKS_PROJECT_ROOT}/demo" --file ./demo.py
tp lakehouse notebooks run "${TP_DATABRICKS_PROJECT_ROOT}/demo" --compute serverless --wait
tp lakehouse jobs list
tp lakehouse jobs upsert-notebook --name daily-demo --notebook-path "${TP_DATABRICKS_PROJECT_ROOT}/demo" --compute serverless --dry-run
tp lakehouse jobs upsert-notebook --name daily-demo --notebook-path "${TP_DATABRICKS_PROJECT_ROOT}/demo" --compute serverless
tp lakehouse jobs run <job_id> --wait
tp lakehouse runs list --job-id <project_job_id>
tp lakehouse runs get <run_id>
tp lakehouse runs output <run_id> --mode compact

Use tp lakehouse runs list --job-id with project-tagged jobs created or managed by tp lakehouse jobs .... For a one-time tp lakehouse notebooks run --wait, inspect the returned run id directly with tp lakehouse runs get <run_id> and tp lakehouse runs output <run_id> instead of listing by the transient internal job id. Use direct databricks ... commands only when CLI auth is explicitly configured in the environment. This can be useful for local agents such as Codex, admin inspection, or temporary troubleshooting. Managed Agent Workspaces should still prefer tp lakehouse.

databricks current-user me
databricks catalogs list
databricks schemas list "$TP_DATABRICKS_CATALOG"
databricks tables list "$TP_DATABRICKS_CATALOG" bronze
databricks workspace list "$TP_DATABRICKS_PROJECT_ROOT"
databricks jobs list

If direct Databricks CLI auth fails but tp lakehouse status works, continue with tp lakehouse; do not switch to PATs or secrets unless the user explicitly approves that auth model.

CLI and MCP execution routes

Intent	Local Agent / Codex	ThinkPlatform Agent
Metadata	`tp lakehouse tables get/list`	Databricks catalog/table MCP system tools
Preview	`tp lakehouse sql preview`	SQL preview system tool, `fresh=false`
Notebook	`tp lakehouse notebooks run --compute ...`	Notebook run system tool, `compute_mode=...`
Job	`tp lakehouse jobs upsert-* --compute ...`	Job upsert system tool, `compute_mode=...`
Cost check	`tp lakehouse costs overview`	`databricks_get_cost_overview`

If a managed Agent lacks the corresponding system tool, report the capability gap. Do not use shell credentials to bypass it. SQL conventions:

CREATE SCHEMA IF NOT EXISTS ${project_catalog}.bronze;
CREATE SCHEMA IF NOT EXISTS ${project_catalog}.silver;
CREATE SCHEMA IF NOT EXISTS ${project_catalog}.gold;

SELECT count(*) AS row_count FROM ${project_catalog}.gold.daily_kpi;
SELECT * FROM ${project_catalog}.gold.daily_kpi LIMIT 20;
SELECT table_name FROM ${project_catalog}.information_schema.tables WHERE table_schema = 'gold';

Always qualify table references with the project catalog. This includes information_schema: use ${project_catalog}.information_schema.tables, not unqualified information_schema.tables, so ThinkPlatform’s project-catalog guard can validate the query.

Prepare data for MLOps

tp mlops data register --name <asset_name> --version <version> --base-path <datalake_or_snapshot_path> --dry-run
tp mlops data list
tp mlops data versions <asset_name>

Verification checklist

Confirm source path/table and output path/table.
Show schema and row counts.
Show a limited sample.
Record data-quality caveats.
Confirm downstream query, import, dashboard, or model training can read the asset.

Get started

CLI

Python SDK

Agent skills

Check context and available resources

Data Lake

Shared DB

Lakehouse and Databricks

CLI and MCP execution routes

Prepare data for MLOps

Verification checklist

​Check context and available resources

​Data Lake

​Shared DB

​Lakehouse and Databricks

​CLI and MCP execution routes

​Prepare data for MLOps

​Verification checklist

Check context and available resources

Data Lake

Shared DB

Lakehouse and Databricks

CLI and MCP execution routes

Prepare data for MLOps

Verification checklist