> ## Documentation Index > Fetch the complete documentation index at: https://docs.thinkplatform.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Choose User guide for browser tasks, Developers for CLI, SDK and external agents, and Administration for organization management. > Confirm identity and organization/project context before operations. Documentation access does not grant ThinkPlatform permissions. > Review side effects, verify changes with a read, and do not blindly retry an operation with an uncertain outcome. > Use documented UI steps for the web app and documented public CLI or SDK interfaces for automation. Follow each page’s prerequisites and authentication requirements. # Data engineering commands > Inspect, transform, and validate data across the Data Lake, shared database, and Lakehouse. These examples assume an authenticated active saved profile with the intended organization and project. Confirm it with `tp context show` before running commands. Add root `--json` before the command when machine parsing needs JSON output; use root `--profile ` only to override the active profile. ## Check context and available resources ```bash theme={"system"} tp auth whoami tp context show tp datalake tree --path data/ tp datalake list --path data/ tp db tables list tp mlops data datastores ``` ## Data Lake ```bash theme={"system"} tp datalake preview --path raw/source.csv tp datalake preview --path raw/source.csv --output /tmp/source.csv tp datalake resolve-uri --path raw/source.csv tp datalake upload-file --local-path ./clean.csv --remote-path processed/clean.csv --dry-run tp datalake upload-file --local-path ./clean.csv --remote-path processed/clean.csv tp datalake upload-dir --local-dir ./outputs --remote-prefix processed/run-001 --dry-run tp datalake move --source-path staging/a.csv --destination-path archive/a.csv --dry-run tp datalake delete --path staging/old.csv --dry-run ``` ## Shared DB ```bash theme={"system"} tp db tables list tp db tables get --table-id tp db tables create --body @table.json --dry-run tp db tables create --body @table.json tp db rows query --table-id --body '{"filters":[],"sort":[],"page":1,"page_size":5}' tp db rows create --table-id --body @row.json --dry-run tp db rows create-many --table-id --body @rows.json --dry-run tp db import apply --file ./import.xlsx --dry-run tp db import apply --file ./import.xlsx ``` ## Lakehouse and Databricks Default to `tp lakehouse` for Databricks/Lakehouse work. These commands call ThinkPlatform's project-scoped Databricks APIs and work in managed Agent Workspaces without requiring direct Databricks CLI auth. Inventory and context: ```bash theme={"system"} tp lakehouse status tp lakehouse catalog show tp lakehouse schemas list tp lakehouse tables list --schema bronze tp lakehouse tables get bronze.example_table tp lakehouse compute list tp lakehouse workspace ls ``` SQL validation and project-scoped writes: ```bash theme={"system"} tp lakehouse tables get bronze.example tp lakehouse sql preview --query "SELECT key, event_time FROM ${TP_DATABRICKS_CATALOG}.bronze.example WHERE event_date >= DATE '2026-01-01' LIMIT 100" tp lakehouse sql preview --file ./query.sql tp lakehouse sql preview --file ./query.sql --fresh tp lakehouse sql execute --statement "CREATE SCHEMA IF NOT EXISTS ${TP_DATABRICKS_CATALOG}.silver" --dry-run tp lakehouse schemas create silver --comment "Cleaned and conformed data" --dry-run ``` Preview uses the shared ThinkPlatform cache by default. Use `--fresh` only after a write when a new Databricks execution is required. In a managed ThinkPlatform Agent, call the corresponding SQL preview MCP system tool with `fresh=false` by default and `fresh=true` only for that explicit read-after-write case. Canvas dashboard handoff: ```bash theme={"system"} tp canvas dashboard scaffold \ --title "Gold Metrics Dashboard" \ --dataset-id gold_metrics \ --query @gold_metrics.sql \ --filter dateRange:period:event_date \ --dry-run tp canvas dashboard schema gold_metrics --definition @page.json tp canvas dashboard query gold_metrics --definition @page.json --filters @filters.json --limit 100 tp canvas dashboard lint --definition @page.json ``` Use this handoff when a dashboard needs global filters across multiple widgets. The dataset query should include named SQL parameters that match each binding target, for example `:event_date_start` and `:event_date_end` for a `dateRange` binding with target `event_date`. Notebook and job workflows: ```bash theme={"system"} tp lakehouse notebooks upsert "${TP_DATABRICKS_PROJECT_ROOT}/demo" --file ./demo.py --dry-run tp lakehouse notebooks upsert "${TP_DATABRICKS_PROJECT_ROOT}/demo" --file ./demo.py tp lakehouse notebooks run "${TP_DATABRICKS_PROJECT_ROOT}/demo" --compute serverless --wait tp lakehouse jobs list tp lakehouse jobs upsert-notebook --name daily-demo --notebook-path "${TP_DATABRICKS_PROJECT_ROOT}/demo" --compute serverless --dry-run tp lakehouse jobs upsert-notebook --name daily-demo --notebook-path "${TP_DATABRICKS_PROJECT_ROOT}/demo" --compute serverless tp lakehouse jobs run --wait tp lakehouse runs list --job-id tp lakehouse runs get tp lakehouse runs output --mode compact ``` Use `tp lakehouse runs list --job-id` with project-tagged jobs created or managed by `tp lakehouse jobs ...`. For a one-time `tp lakehouse notebooks run --wait`, inspect the returned run id directly with `tp lakehouse runs get ` and `tp lakehouse runs output ` instead of listing by the transient internal job id. Use direct `databricks ...` commands only when CLI auth is explicitly configured in the environment. This can be useful for local agents such as Codex, admin inspection, or temporary troubleshooting. Managed Agent Workspaces should still prefer `tp lakehouse`. ```bash theme={"system"} databricks current-user me databricks catalogs list databricks schemas list "$TP_DATABRICKS_CATALOG" databricks tables list "$TP_DATABRICKS_CATALOG" bronze databricks workspace list "$TP_DATABRICKS_PROJECT_ROOT" databricks jobs list ``` If direct Databricks CLI auth fails but `tp lakehouse status` works, continue with `tp lakehouse`; do not switch to PATs or secrets unless the user explicitly approves that auth model. ## CLI and MCP execution routes | Intent | Local Agent / Codex | ThinkPlatform Agent | | - | - | - | | Metadata | `tp lakehouse tables get/list` | Databricks catalog/table MCP system tools | | Preview | `tp lakehouse sql preview` | SQL preview system tool, `fresh=false` | | Notebook | `tp lakehouse notebooks run --compute ...` | Notebook run system tool, `compute_mode=...` | | Job | `tp lakehouse jobs upsert-* --compute ...` | Job upsert system tool, `compute_mode=...` | | Cost check | `tp lakehouse costs overview` | `databricks_get_cost_overview` | If a managed Agent lacks the corresponding system tool, report the capability gap. Do not use shell credentials to bypass it. SQL conventions: ```sql theme={"system"} CREATE SCHEMA IF NOT EXISTS ${project_catalog}.bronze; CREATE SCHEMA IF NOT EXISTS ${project_catalog}.silver; CREATE SCHEMA IF NOT EXISTS ${project_catalog}.gold; SELECT count(*) AS row_count FROM ${project_catalog}.gold.daily_kpi; SELECT * FROM ${project_catalog}.gold.daily_kpi LIMIT 20; SELECT table_name FROM ${project_catalog}.information_schema.tables WHERE table_schema = 'gold'; ``` Always qualify table references with the project catalog. This includes `information_schema`: use `${project_catalog}.information_schema.tables`, not unqualified `information_schema.tables`, so ThinkPlatform's project-catalog guard can validate the query. ## Prepare data for MLOps ```bash theme={"system"} tp mlops data register --name --version --base-path --dry-run tp mlops data list tp mlops data versions ``` ## Verification checklist * Confirm source path/table and output path/table. * Show schema and row counts. * Show a limited sample. * Record data-quality caveats. * Confirm downstream query, import, dashboard, or model training can read the asset. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.