3月 23, 2022

Data Management OSS (Object Storage Service) migrating to Direct-to-S3 approach

ForgeToS3

As part of our commitment to innovate and improve customer services, Autodesk released a new set of Data Management API endpoints to allow direct upload and download from the underlying AWS S3 bucket. The primary impact of this direct-to-S3 approach is performance.

Our testing and validation continue to show a significant increase in upload and download speeds, especially for large files. 

How? When an object is uploaded or downloaded from an OSS bucket, Autodesk uses AWS S3 storage.

The current binary transfer approach moves the data through our proxy, while the new direct-to-S3 approach enables you to directly access the underlying S3 bucket and object.   

Direct binary streaming data management endpoints to be deprecated September 30, 2022

 Autodesk will deprecate the current binary transfer streaming endpoints approach of uploading and downloading files directly to and from our proxy (developer.api.autodesk.com) on September 30, 2022, and such transfer will no longer be allowed after that date.

You will be required to move your existing application(s) utilizing impacted Autodesk APIs to the new direct-to-S3 approach by September 30, 2022. 

See below for current information on impacted API endpoints and stay tuned for additional details to be released soon. 

The following APIs are affected:  

  • Model Derivative: The GET Derivatives endpoint will change to allow download of binary files. A new endpoint signature will be released soon and announced with a different deprecation date.

  • Reality Capture: More details soon.  

  

The following APIs are not affected, but your application may have the following cross dependencies:    

  • Design Automation: OSS is commonly used as storage for the Design Automation workitems input and output. No change is happening with Design Automation. As explained in the next section (“Direct-to-S3 approach for Data Management OSS”), if a workitem is saved to OSS, the application will need to finalize the upload inside the OnComplete callback.   

  • Data Management – Projects, Folders & Files: Applications that upload and download files will need to use signed S3 URLs.   

  • Autodesk Construction Cloud & BIM360: More details coming soon.

  • Viewer: More details coming soon.

  

The following APIs are NOT affected:  

  • Authentication  

  • Data Visualization  

  • Token Flex  

  • Viewer  

  • Webhooks  

  • Cost  

  • Assets  

 

Direct-to-S3 approach for Data Management OSS  

To upload and download files, applications must generate a signed URL, then upload or download the binary. Here are the steps (pseudo code):   

Upload  

  1. Calculate the number of parts of the file to upload  

    1. Note: Each uploaded part except for the last one must be at least 5MB  

  2. Generate up to 25 URLs for uploading specific parts of the file using the GET buckets/:bucketKey/objects/:objectKey/signeds3upload?firstPart=<index of first part>&parts=<number of parts> endpoint  

    1. The part numbers start with 1  

    2. For example, to generate upload URLs for parts 10 through 15, set firstPart to 10 and parts to 6  

    3. This endpoint also returns an uploadKey that is used later to request additional URLs or to finalize the upload  

  3. Upload remaining parts of the file to their corresponding upload URLs  

    1. Consider retrying (for example, with an exponential backoff) individual uploads when the response code is 100-199, 429, or 500-599  

    2. If the response code is 403, the upload URLs have expired; go back to step #2  

    3. If you have used up all the upload URLs and there are still parts that must be uploaded, go back to step #2  

  4. Finalize the upload using the POST buckets/:bucketKey/objects/:objectKey/signeds3upload endpoint, using the uploadKey value from step #2    

Download  

  1. Generate a download URL using the GET buckets/:bucketKey/objects/:objectName/signeds3download endpoint  

  1. Use the new URL to download the OSS object directly from AWS S3  

    1. Consider retrying (for example, with an exponential backoff) the download when the response code is 100-199, 429, or 500-599  

 

The signed URLs have a default expiration time of 2 minutes (*longer expiration times can be set using the minutesExpiration param up to 60 minutes). It is important that the download or upload process starts before the expiration. In case of single or multi-part upload, you may request signed URLs again. A set of steps for uploading and downloading can be found in our Step-by-Step Documentation. 

In our next series of blog posts, we will be sharing with all of you a couple of samples the Developer Advocates are currently working on, keep an eye for the Node.js, .Netcore and .NET ones coming very soon.

Thank you again for all your business and please feel free to reach out to us through forge.help@autodesk.com

 

*DISCLAIMER When generating signed URLs, it’s important to use the smallest possible expiration time, in order to avoid longer access in case of exposure of the URL.

Posted by

Jaime Rosales

Senior Developer Advocate

Jaime Rosales is a Senior Developer Advocate with Autodesk Forge. He’s part of the team that helps partners/customers create new products and transition to the cloud with the use of Autodesk’s new Platform - Forge. He joined Autodesk in 2011 through the acquisition of Horizontal Systems the company that developed the cloud-based collaboration systems—now known as BIM 360 Glue the...

Related Posts