- 05 Oct 2021
- 11 Minutes to read
- Print
- DarkLight
- PDF
Asset Directory API
- Updated on 05 Oct 2021
- 11 Minutes to read
- Print
- DarkLight
- PDF
The Asset Directory API provides simple RESTful access to all asset storage capabilities. Like feeds, asset directories expose their APIs through the URL /endpoints/«AssetDirectoryName»/...
Asset Directories are represented internally in ProGet as Feeds, so Feed API keys can also be used to authenticate for Asset Directories.
Content API
The Asset Content API is accessible under /endpoints/«AssetDirectoryName»/content/..., and provides the primary access point for working with assets. This endpoint is designed to function like a standard web server, so hosted assets can be accessed by simple GET requests with support for browser caching.
Get Asset Endpoint
GET .../content/«path_to_asset»
Gets the asset at the specified path, returning it as content. This endpoint returns a status of 200 (success), 304 (success, not modified), 404 (asset not found), 401 (auth required), 403 (access denied).
On success, the asset is returned as content.
Test for Asset Endpoint
HEAD .../content/«path_to_asset»
Returns the headers only for the GET endpoint. This can be used to determine if an asset exists. This endpoint returns a status of 200 (success), 304 (success, not modified), 404 (asset not found), 401 (auth required), 403 (access denied).
Note that as this is a HEAD request, there is no content in the response.
Create or Replace Asset Endpoint
POST .../content/«path_to_asset»
Creates a new asset (or overwrites an existing asset) at the specified path using the request content as the asset. This endpoint returns a status of 201 (success), 401 (auth required), 403 (access denied). If specified, the Content-Type header will be stored for the asset.
On success, the asset is saved to the asset directory.
Create Asset Endpoint
PUT .../content/«path_to_asset»
Creates a new asset (but will not overwrite an existing asset) at the specified path using the request content as the asset. This endpoint returns a status of 201 (success), 400 (asset already exists), 401 (auth required), 403 (access denied). If specified, the Content-Type header will be stored for the asset.
On success, the asset is saved to the asset directory.
Replace Asset Endpoint
PATCH .../content/«path_to_asset»
Overwrites an existing asset at the specified path using the request content as the asset. This endpoint returns a status of 201 (success), 404 (asset not found), 401 (auth required), 403 (access denied). If specified, the Content-Type header will be stored for the asset.
On success, the existing asset is replaced.
Delete Asset Endpoint
DELETE .../content/«path_to_asset»
Deletes the asset at the specified path. The path must refer to an invidual asset and not a directory. It is not considered an error to delete a file that does not exist. This endpoint returns a status of 200 (success), 400 (path refers to a directory), 401 (auth required), 403 (access denied).
On success, the asset is deleted from the asset directory.
Directory API
The Asset Directory API is accessible under /endpoints/«AssetDirectoryName»/dir/..., and extends the Content API by providing explicit endpoints for working with directories.
List Directory Endpoint
GET .../dir/«path»?recursive=«true/false»
Returns all assets and directories in the specified path as an array of JSON items. When recursive is false or not specified, only items contained in the specified path are returned. When recursive is true, all items in subdirectories are also returned. This endpoint returns a status of 200 (success), 404 (directory not found), 401 (auth required), 403 (access denied).
Data Specification
Asset Item
Property | Format |
---|---|
name | A string containing the local name of the asset. This property does not include the full path. |
parent | A string containing the full path of the parent directory of the asset. This property does not include the name of the asset itself. This property may be omitted if the asset is contained in the directory root. |
type | A string containing either the Content-Type of the the asset, or the literal text dir if the item represents a subdirectory. |
content | A string containing a full URL where this file can be downloaded. This property is not present if the item is a subdirectory. |
created | A string containing the UTC date of the original creation time of the item in ISO 8601 format (yyyy-MM-ddThh:mm:ss). |
modified | A string containing the UTC date of the last time of the item was updated in ISO 8601 format (yyyy-MM-ddThh:mm:ss). This property is omitted if the item represents a subdirectory. |
size | A number specifying the number of bytes in size of the asset item. This property is omitted if the item represents a subdirectory. |
sha1 | A string containing the SHA1 hash of the asset item. This property is omitted if the item represents a subdirectory. |
On success, a JSON array of the above objects is returned.
Create Directory Endpoint
POST .../dir/«path»
Creates a directory at the specified path. If any of the parent directories do not exist, they will be created as well. It is not an error if the directory already exists. This endpoint returns a status of 201 (success), 401 (auth required), 403 (access denied).
Delete API
The Asset Delete API is accessible under /endpoints/«AssetDirectoryName»/delete/..., and further extends the Content API by providing explicit endpoints for deleting items without using an HTTP DELETE request.
Delete Item Endpoint
POST .../delete/«path»?recursive=«true/false»
Deletes the asset or directory at the specified path. When recursive is false or not specified and the path refers to a directory, the directory will only be deleted if it is empty. When recursive is true, the item and all of its contents (if it is a directory) will be deleted. It is not an error if the directory does not exist. This endpoint returns a status of 200 (success), 400 (directory not empty and recursive=false), 401 (auth required), 403 (access denied).
Metadata API
The Asset Metdata API is accessible under /endpoints/«AssetDirectoryName»/metadata/..., and provides endpoints for reading or updating metadata on an asset or subdirectory. This API was added in ProGet v6.0.0.
Get Asset Metadata Endpoint
GET .../metadata/«path_to_asset»
Returns metadata for the specified asset file/directory as a JSON object with the following properties:
Property | Format |
---|---|
name | A string containing the local name of the asset. This property does not include the full path. |
parent | A string containing the full path of the parent directory of the asset. This property does not include the name of the asset itself. This property may be omitted if the asset is contained in the directory root. |
type | A string containing either the Content-Type of the the asset, or the literal text dir if the item represents a subdirectory. |
content | A string containing a full URL where this file can be downloaded. This property is not present if the item is a subdirectory. |
created | A string containing the UTC date of the original creation time of the item in ISO 8601 format (yyyy-MM-ddThh:mm:ss). |
modified | A string containing the UTC date of the last time of the item was updated in ISO 8601 format (yyyy-MM-ddThh:mm:ss). This property is omitted if the item represents a subdirectory. |
size | A number specifying the number of bytes in size of the asset item. This property is omitted if the item represents a subdirectory. |
sha1 | A string containing the SHA1 hash of the asset item. This property is omitted if the item represents a subdirectory. |
userMetadata | An object listing key/value pairs of user-defined metadata for the item. |
cacheHeader | An object containing the type of caching used and the value configured. See cache header settings for more information. |
Update Asset Metadata Endpoint
POST .../metadata/«path_to_asset»
Updates the Content-Type or the user-defined metadata of an asset. The request body must have a Content-Type of application/json, and must contain a JSON object with the following properties:
Property | Format |
---|---|
type | A string containing the new Content-Type of the the asset. Ignored for directories. If not specified, the Content-Type is not updated. |
userMetadata | An object listing updated key/value pairs of user-defined metadata for the item. |
userMetadataUpdateMode | A string containing one of: update (create/update properties) or replace (create/update properties and delete missing values) |
cacheHeader | An object containing two properties, type and value . If not specified, the Cache Header is not updated. (ex: { 'type': 'TTL', 'value': 30 } ). |
Cache Header settings
See client-side caching for more information.
Type | Description | Value |
---|---|---|
Inherit | Inherit from parent folder | Value is not needed |
NoCache | Do not cache this item | Value is not needed |
TTL | Length in seconds to cache items | Integer value (ex: 30 ) |
Custom | Use a custom Cache-Control header with the specified value | String value (ex: public, max-age=604800, immutable ) |
Export API
The Asset Export API is accessible under /endpoints/«AssetDirectoryName»/export/..., and provides endpoints for downloading batches of assets at once.
Export Directory Endpoint
GET .../export/«path»?format=«zip/tgz»&recursive=«true/false»
Returns the contents of the specified directory as either a ZIP or a TGZ archive. The format argument may be either zip (for a zip file) or tgz for a GZipped tar file. When recursive is false or not specified, only items contained directly in the specified path are included. When recursive is true, the archive will contain all subdirectories as well. This endpoint returns a status of 200 (success), 404 (directory not found), 400 (invalid format), 401 (auth required), 403 (access denied).
Import Directory Endpoint
POST .../import/«path»?format=«zip/tgz»&overwrite=«true/false»
Adds the contents of the uploaded archive to the specified path. The format argument may be either zip (for a zip file) or tgz for a GZipped tar file. When overwrite is false or not specified, items already in the asset directory will never be overwritten. When overwrite is true, items in the asset directory will be overwritten. If the specified directory does not exist, it will be created. This endpoint returns a status of 200 (success), 400 (invalid format), 401 (auth required), 403 (access denied).
Multipart Asset Upload
For very large assets, an upload can be performed using multiple requests using the multipart upload API. This endpoint functions in the same way as the standard Create Asset Endpoint, except requires some additional query string arguments. A multipart upload consists of one or more chunk uploads followed by a final POST to indicate that the upload is complete.
Note: Due to technical limitations, multipart uploads are currently not supported for cloud-based package stores. ProGet may extend support for multipart uploads to these in a later version, but currently any attempt to initiate a multipart upload with an S3/Azure backed package store will result in an HTTP 400 error.
Multipart Upload: Upload Chunk
POST .../content/«path_to_asset»?multipart=upload&id=«uuid»&index=«partIndex»&offset=«byteOffset»&totalSize=«byteSize»&partSize=«partSize»&totalParts=«partCount»
Initiates a new multipart upload or uploads the next part in a previously started multipart upload. All arguments are required:
- id - Unique identifier for the upload. It is the client's responsibility to generate this. It does not need to be a GUID, though a GUID is perfectly valid.
- index - Zero-based index of the part being uploaded.
- offset - Zero-based byte offset of the part being uploaded.
- totalSize - Size of the entire asset being uploaded in bytes. This must be exactly the sum of all individual uploaded part sizes.
- partSize - Size of the part being uploaded.
- totalParts - Total number of parts that will be uploaded for the entire asset.
This endpoint returns a status of 200 (success), 401 (auth required), 403 (access denied), 400 (invalid arguments such as index/offset out of range, overlapping parts, invalid size).
Multipart Upload: Complete Upload
POST .../content/«path_to_asset»?multipart=complete&id=«uuid»
Completes a multipart upload after all individual parts have been uploaded. The asset will not be added to the asset directory until the upload has been marked as complete with this endpoint.
This endpoint returns a status of 200 (success), 401 (auth required), 403 (access denied), 400 (invalid arguments such as index/offset out of range, overlapping parts, invalid size).
On success, the asset is saved to the asset directory.
Note: Orphaned parts will be deleted during the Feed Cleanup scheduled task. Orphaned parts can occur when a client uploads one or more parts but never completes the upload.
Multipart Upload: PowerShell Script
This PowerShell script can be used to automatically perform a multipart upload if necessary. Feel free to use it as-is, or as a starting point for your own solution:
<#
.Synopsis
Transfers a file to a ProGet asset directory.
.Description
Transfers a file to a ProGet asset directory. This function performs automatic chunking
if the file is larger than a specified threshold.
.Parameter FileName
Name of the file to upload from the local file system.
.Parameter EndpointUrl
Full URL of the ProGet asset directory's API endpoint. This is typically something like http://proget/endpoints/<directoryname>
.Parameter AssetName
Full path of the asset to create in ProGet's asset directory.
.Parameter ChunkSize
Uploads larger than this value will be uploaded using multiple requests. The default is 5 MB.
.Example
Upload-ProGetAsset -FileName C:\Files\Image.jpg -AssetName images/image.jpg -EndpointUrl http://proget/endpoints/MyAssetDir
#>
function Upload-ProGetAsset {
param(
[Parameter(Mandatory = $true)]
[string] $fileName,
[Parameter(Mandatory = $true)]
[string] $endpointUrl,
[Parameter(Mandatory = $true)]
[string] $assetName,
[int] $chunkSize = 5 * 1024 * 1024
)
function CopyMaxBytes {
param(
[System.IO.Stream] $source,
[System.IO.Stream] $target,
[int] $maxBytes,
[long] $startOffset,
[long] $totalSize
)
$buffer = [Array]::CreateInstance([System.Byte], 32767)
$totalBytesRead = 0
while ($true) {
$bytesRead = $source.Read($buffer, 0, [Math]::Min($maxBytes - $totalBytesRead, $buffer.Length))
if($bytesRead -eq 0) {
break
}
$target.Write($buffer, 0, $bytesRead)
$totalBytesRead += $bytesRead
if($totalBytesRead -ge $maxBytes) {
break
}
$overallProgress = $startOffset + $totalBytesRead
Write-Progress -Activity "Uploading $fileName..." -Status "$overallProgress/$totalSize" -PercentComplete ($overallProgress / $totalSize * 100)
}
}
if(-not $endpointUrl.EndsWith('/')) {
$endpointUrl += '/';
}
$targetUrl = $endpointUrl + 'content/' + [Uri]::EscapeUriString($assetName.Replace('\', '/'))
$fileInfo = (Get-ChildItem -Path $fileName)
if($fileInfo.Length -le $chunkSize) {
Invoke-WebRequest -Method Post -Uri $targetUrl -InFile $fileName
} else {
$sourceStream = New-Object -TypeName System.IO.FileStream -ArgumentList ($fileName, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::Read, 4096, [System.IO.FileOptions]::SequentialScan)
try {
$fileLength = $sourceStream.Length
$remainder = [long]0
$totalParts = [Math]::DivRem([long]$fileLength, [long]$chunkSize, [ref]$remainder)
if($remainder -ne 0) {
$totalParts++
}
$uuid = (New-Guid).ToString("N")
for($index = 0; $index -lt $totalParts; $index++) {
$offset = $index * $chunkSize
$currentChunkSize = $chunkSize
if($index -eq ($totalParts - 1)) {
$currentChunkSize = $fileLength - $offset
}
$req = [System.Net.WebRequest]::CreateHttp("${targetUrl}?multipart=upload&id=$uuid&index=$index&offset=$offset&totalSize=$fileLength&partSize=$currentChunkSize&totalParts=$totalParts")
$req.Method = 'POST'
$req.ContentLength = $currentChunkSize
$req.AllowWriteStreamBuffering = $false
$reqStream = $req.GetRequestStream()
try {
CopyMaxBytes -source $sourceStream -target $reqStream -maxBytes $currentChunkSize -startOffset $offset -totalSize $fileLength
}
finally {
if($reqStream -ne $null) {
$reqStream.Dispose()
}
}
$response = $req.GetResponse()
try {
}
finally {
if($response -ne $null) {
$response.Dispose()
}
}
}
Write-Progress -Activity "Uploading $fileName..." -Status "Completing upload..." -PercentComplete -1
$req = [System.Net.WebRequest]::CreateHttp("${targetUrl}?multipart=complete&id=$uuid")
$req.Method = 'POST'
$req.ContentLength = 0
$response = $req.GetResponse()
try {
}
finally {
if($response -ne $null) {
$response.Dispose()
}
}
}
finally {
if($sourceStream -ne $null) {
$sourceStream.Dispose()
}
}
}
}