Step-by-Step: Importing Files into LumiDB Using the API

Do not index

At LumiDB, we’re building a database for reality capture data that makes handling big point cloud datasets easy, accessible and fun. Request access to the alpha version to try it out!

A while ago, I wrote about LumiDB’s simple Runtime API. But it wouldn’t be much of a database unless you could put some data into it. This post shows you how to import a bunch of .laz files into a LumiDB table, by walking through our equally simple import API.

The import process involves the following steps:

(Generate presigned URLs for input files)

Create a manifest

Call the import API

Poll for import status

Query imported data (optional)

Prerequisites

You’ll need:

An API key with import privileges.

Input files (e.g. .laz) either directly available via HTTP GET, or in S3 for generating pre-signed URLs

The target server's public base URL, e.g. https://lumidb.example.com.

Step 1: Generate Presigned URLs

Each input file must be accessible to LumiDB via an HTTP(S) URL. If using AWS S3, we recommend you generate presigned URLs. For example, you can use the AWS CLI to do this:

aws s3 presign s3://your-bucket-name/path/to/file.laz --expires-in 3600

Step 2: Create a Manifest

The manifest is a JSON object describing the import. Here's an example:

{
  "table_name": "example_dataset",
  "table_proj": "EPSG:3879",
  "inputs": [
    {
      "src": "https://example.com/path/to/fileA.laz?AWSAccessKeyId=...",
      "date": "2021-02-03",
      "proj": "EPSG:3879",
      "license": "Example License"
    },
    {
      "src": "https://example.com/path/to/fileB.laz?AWSAccessKeyId=...",
      "date": "2021-06-22",
      "proj": "EPSG:3879",
      "license": "Example License"
    }
  ]
}

The table_proj field defines the coordinate reference system for the table. Each individual file also has a respective proj field defining the input projection of the file. Each file's data will be converted into the table's native projection. Both fields accept either an EPSG code or a PROJ string. This means that you can combine input files from different projections into the same table. It makes sense to choose the table’s native projection so that it matches the projection in which you typically want the queries to return the data, to avoid the conversion overhead during queries.

In a manifest, the table_name, table_proj and each input file's src and proj fields are mandatory. In addition, each input file can have arbitrary user-defined metadata properties, which will be available through the query API after import, and can be utilized by LumiDB's metadata queries. In the example above, we define date and license, but the user may as well define capture_weekday, producer or anything else.

Step 3: Launch the Import Process

The heavy lifting is done by the /api/tables/import HTTP endpoint. Send a POST request to the endpoint with the manifest:

curl -X POST https://lumidb.example.com/api/tables/import \
  -H "Authorization: apikey YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d @manifest.json

The response will contain a table_version ID which is then used to poll the import progress:

{
    "table_version": "123e4567-e89b-12d3-a456-426614174000"
}

Step 4: Poll for Import Status

To check the progress of the import, poll the /api/tables/import_status/{table_version} endpoint:

curl https://lumidb.example.com/api/tables/import_status/123e4567-e89b-12d3-a456-426614174000 \
  -H "Authorization: apikey YOUR_API_KEY"

The response will be one of the following:

{ "status": "queued" }
{ "status": "processing", "progress": 42.0 }
{ "status": "ready" }
{ "status": "failed", "error_msg": "..." }

Where progress is a percentage between 0 and 100. You can poll every few seconds until the status is either ready or failed.

Step 5: Query the Imported Data

Once the table is ready, you can proceed to query it, e.g. using the NPM library:

// Query points from a table
const response = await lumidb.query({
    tableName: tables[0].tableName,
    queryBoundary: {
        Polygon: [
            // polygon boundary
            [
                [
                    [2777342, 8438905],
                    [2777342, 8439315],
                    [2777573, 8439315],
                    [2777573, 8438905],
                    [2777342, 8438905],
                ],
            ],
            // optional z range
            [0, 100],
        ],
    },
    queryProj: "EPSG:3857",
    maxPoints: 5_000_000,
    maxDensity: 10.0,
    sourceFileFilter: null, // see below
    classFilter: null,
    outputType: "threejs", // or "laz" for exporting to other software
});

NOTE: If you re-imported a table, that is, launched the import using an existing table name, the previous version of the table will be available for queries until the new import process is ready, after which the table name pointer will be atomically changed to point to the newly imported version.

It’s that simple! If you’d like to get your hands on an API key and try it out for yourself, sign up for LumiDB Alpha! LumiDB is currently in alpha and available for early adopters!