10 Aug 2020

Accessing design metadata without the Viewer

Design metadata

Sometimes, developers need to be able to access the metadata of their designs in Autodesk Forge without necessarily using the viewer. In this post we will take a look at some of the options the platform offers in this regard.

Option #1: HTTP endpoint

The Model Derivative service provides an endpoint for retrieving metadata of a specific design in form of a JSON: GET :urn/metadata/:guid/properties. The JSON responses are easy to parse, and the endpoint can also be configured to only return properties of a single object with specific ID, making the requests faster and more efficient.

However, before deciding to use this option in your application, consider the following characteristics of this endpoint:

  • JSON responses may be too verbose for complex designs; if the response exceeds 20MB, an additional URL parameter forceget must be used in order for the request to be completed
  • Compared to other approaches, the output is less granular; for example, measurement units are not stored separately but "baked" into property values

Option #2: Sqlite database

When processing your designs, the Model Derivative service stores all the property metadata in a sqlite file. You can find it in the derivative manifest (using the GET :urn/manifest endpoint) as an asset with mime type application/autodesk-db, and with role set to Autodesk.CloudPlatform.PropertyDatabase, for example:

{
  "urn": "dXJuOmZvbw",
  "derivatives": [
    {
      "children": [
        // ...
        {
          "urn": "urn:adsk.viewing:fs.file:dXJuOmZvbw/output/Resource/model.sdb",
          "role": "Autodesk.CloudPlatform.PropertyDatabase",
          "mime": "application/autodesk-db",
          "guid": "6fac95cb-af5d-3e4f-b943-8a7f55847ff1",
          "type": "resource",
          "status": "success"
        }
        // ...
      ]
    }
  ]
}

To download this asset, simply use the GET :urn/manifest/:derivativeurn endpoint, passing in the URN of the property database (note: url-encoded, not base64-encoded). For the example above, the endpoint would look like this: https://developer.api.autodesk.com/modelderivative/v2/designdata/dXJuOmZvbw/manifest/urn%3Aadsk.viewing%3Afs.file%3AdXJuOmZvbw%2Foutput%2FResource%2Fmodel.sdb.

Now, when you peek inside the sqlite file, you'll notice a couple of tables with strange names like _objects_attr_objects_eav_objects_id_objects_val, etc. These tables form a structure called entity-attribute-value that we will describe at the end of this blog post.

Option #3: Viewer-optimized database

Forge Viewer uses a different representation of the property database (mainly for performance reasons), stored in several *.json.gz files with similar naming as the tables in the sqlite database, for example, objects_attrs.json.gzobjects_avs.json.gzobjects_ids.json.gz, or objects_vals.json.gz. The relationships between the data in these compressed JSON files is explained in the next section.

Note that the .json.gz files are not part of the derivative manifest described earlier. Instead, they are part of the internal format used by the Forge Viewer called SVF. You can identify and download these assets with tools like forge-convert-utils (specifically, the SvfDownloader class). The same library also provides a simple PropDbReader class that can be used to parse the .json.gz files and make basic queries over their data. A sample code using this class is available in https://github.com/petrbroz/forge-convert-utils/blob/develop/samples/remote-svf-props.js.

Entity-attribute-value structure

Whether you're looking at the sqlite database or at the .json.gz files, the structure is basically the same - a collection of entity-attribute-value triplets (EAVs) linking together objects and their properties in the following way:

  • entity is an index into an "entity" table (_objects_id or objects_ids.json.gz) containing information such as:
    • object ID (also known as "dbId")
    • external ID (ID persistent across versions of the same design; different for each type of design format)
  • attribute is an index into an "attribute" table (_objects_attr or objects_attrs.json.gz) containing information such as:
    • property name
    • property category
    • property type and units,
    • property flags
  • value is an index to a table of property values (_objects_val or objects_vals.json.gz)

 

Here's a diagram of how the 5 json files (objects_ids.json.gz, objects_attrs.json.gz, objects_vals.json.gz, objects_avs.json.gz, objects_offs.json.gz), join together to form our properties meta-data:
 

json-gz-Entity-attribute-value

And here is a diagram of the same data, but in SQLite tables:

Entity-attribute-value

With this information we can start slicing & dicing the sqlite database in any way we need. For example, to get the IDs of all objects containing the word "Concrete" in their "Material" property, we could use the following query:

SELECT _objects_id.id AS dbId, _objects_id.external_id AS externalId, _objects_attr.display_name AS propName, _objects_val.value AS propValue
FROM _objects_eav
    INNER JOIN _objects_id ON _objects_eav.entity_id = _objects_id.id
    INNER JOIN _objects_attr ON _objects_eav.attribute_id = _objects_attr.id
    INNER JOIN _objects_val ON _objects_eav.value_id = _objects_val.id
WHERE propName = "Material" AND propValue LIKE "%Concrete%"

Related Article