Path Foundation serving API

Stay organized with collections Save and categorize content based on your preferences.

This document describes the Application Programming Interface (API) for Path Foundation when deployed as an HTTPS service endpoint, referred to as the service in this document.

Overview

The serving source code for Path Foundation can be built and hosted on any API management system, but it's specially designed to take advantage of Vertex AI prediction endpoints. Therefore, it conforms to Vertex AI's required API signature and implements a predict method.

The service is designed to support micro batching, not to be mistaken with batch jobs. For every pathology image patch in the request, if the processing is successful, the service returns a one-dimensional embedding vector in the same order as the image patches in the request. Refer to the sections on API request, response, and micro batching for details.

You can provide digital pathology image patches to the service either directly within the request (inlined) or by providing a reference to their location. Inlining the images in the request is not recommended for large-scale productions; read more. When using data storage links the service expects corresponding OAuth 2.0 bearer tokens to retrieve the data on your behalf. For detailed information on constructing API requests and the different ways to provide image data, refer to the API request section.

When given DICOM images from a DICOM store, the service expects the underlying DICOM storage system to conform to HAI-DEF DICOM store requirements. Furthermore, the service expects the pathology data to meet more detailed requirements.

To invoke the service, consult the request section, compose a valid request JSON and send a POST request to the endpoint. The following script shows a sample cURL command which is initialized to invoke our research endpoint. You can deploy Path Foundation using Model Garden and change LOCATION, PROJECT_ID and ENDPOINT_ID to target your endpoint:

LOCATION="us-central1"
PROJECT_ID="hai-cd3-foundations"
ENDPOINT_ID="162"
REQUEST_JSON="path/to/your/request.json"

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/endpoints/${ENDPOINT_ID}:predict" \
-d "@${REQUEST_JSON}"

Request

An API request can include multiple instances, each conforming to this schema. Note that this schema is based on Vertex AI PredictSchemata standard and is a partial OpenAPI specification. The complete JSON request has the following structure:

{
  "instances": [
    {...},
    {...}
  ]
}

Each request instance can include an optional extension dictionary that you can use to control certain service behaviour; read more about extensions.

The service accepts digital pathology image patches in two ways:

• Directly within the HTTPS request: You can include image data as base64-encoded bytes using the raw_image_bytes JSON field; read more about inlined images.

• Indirectly via storage links: You can provide links to the images stored in GCS using the image_file_uri JSON field, or you can use dicom_path to point to DICOM images stored in DICOMweb storage; read more about DICOM requirements.

To illustrate these methods, the following example JSON request shows raw_image_bytes, image_file_uri, and dicom_path all in one request. In a real-world scenario, you'll typically only use one of these options for all images within a single request:

{
  "instances": [
    {
      "raw_image_bytes": "your base 64 encoded image bytes",
      "patch_coordinates": [
        {
          "x_origin": 0,
          "y_origin": 0,
          "width": 224,
          "height": 224
        }
      ],
      "extensions": {
        "transform_imaging_to_icc_profile": "sRGB",
        "require_patches_fully_in_source_image": true
      }
    },
    {
      "image_file_uri": "gs://your-bucket/path/to/image.png",
      "bearer_token": "your-bearer-token",
      "patch_coordinates": [
        {
          "x_origin": 0,
          "y_origin": 0,
          "width": 224,
          "height": 224
        }
      ],
      "extensions": {
        "transform_imaging_to_icc_profile": "sRGB"
      }
    },
    {
      "dicom_path": {
        "series_path": "https://dicomweb-store.uri/studies/1.2.3.4.5.6.7.8.9/series/1.2.3.4.5.6.7.8.10",
        "instance_uids": ["1.2.3.4.5.6.7.8.11", "1.2.3.4.5.6.7.8.12"],
      }
      "bearer_token": "your-bearer-token",
      "patch_coordinates": [
        {
          "x_origin": 0,
          "y_origin": 0,
          "width": 224,
          "height": 224
        },
        {
          "x_origin": 500,
          "y_origin": 500,
          "width": 224,
          "height": 224
        }
      ],
      "extensions": {
        "transform_imaging_to_icc_profile": "sRGB"
      }
    }
  ]
}

Inlined images

You can inline the images in the API request as a base64-encoded string in the raw_image_bytes JSON field. However, keep in mind most API management systems enforce a limit on the maximum size of the request payloads. When Path Foundation is hosted as a Vertex AI Prediction endpoint, Vertex AI quotas apply.

To optimize the request size, you should compress the images using common image compression codecs. If you require lossless compression, use PNG encoding. If lossy compression is acceptable, use JPEG encoding.

Here is a code snippet for converting compressed JPEG image files from local file system into a base64-encoded string:

import base64

def encode_file_bytes(file_path: str) -> str:
  """Reads a file and returns its contents as a base64-encoded string."""

  with open(file_path, 'rb') as imbytes:
    return base64.b64encode(imbytes.read())

Another code snippet for converting uncompressed image bytes into a lossless PNG format and then converting it into a base64-encoded string:

import base64
import io
import numpy as np
import PIL.Image

def convert_uncompressed_image_bytes_to_base64(image: np.ndarray) -> str:
  """Converts an uncompressed image array to a base64-encoded PNG string."""

  with io.BytesIO() as compressed_img_bytes:
    with PIL.Image.fromarray(image) as pil_image:
      pil_image.save(compressed_img_bytes, 'png')
    return base64.b64encode(compressed_img_bytes.getvalue())

Extensions

The API request supports an optional freeform dictionary for each instance. You can think of the extensions as server side configurations that you can control in your requests. The current list of supported extensions are as follows:

Response

An API response can include multiple predictions that correspond to the order of the instances in the request. Each prediction conforms to this schema. Note that this schema is based on Vertex AI PredictSchemata standard and is a partial OpenAPI specification. The complete JSON request has the following structure:

{
  "predictions": [
    {
      // ... content of first prediction
    },
    {
      // ... content of second prediction
    }
  ],
  "deployedModelId": "model-id",
  "model": "model",
  "modelVersionId": "version-id",
  "modelDisplayName": "model-display-name",
  "metadata": {
    // ... content of metadata
  }
}

Each request instance can independently succeed or fail. When succeeded, the corresponding prediction JSON starts with a result field and when failed with an error. Here is an example of a response to a request with two instances where the first one has succeeded and the second one failed:

{
  "predictions": [
    {
      "result": {
        "patch_embeddings": [
          {
            "patch_coordinate": {
              "x_origin": 100,
              "y_origin": 200,
              "width": 224,
              "height": 224
            },
            "embedding_vector": [ 0.1, 0.2, 0.3]
          }
        ]
      }
    },
    {
      "error": {
        "code": "INCOMPATIBLE_PATCH_DIMENSIONS_ERROR",
        "description": "Some actionable text."
      }
    }
  ],
  "deployedModelId": "model-id",
  "model": "model",
  "modelVersionId": "version-id",
  "modelDisplayName": "model-display-name",
  "metadata": {
    // ... content of metadata
  }
}

Error codes

Micro batching

The API request supports micro batching. You can request embeddings for multiple image patches using different instances within the same JSON request:

{
  "instances": [
    {...},
    {...}
  ]
}

Furthermore, within one request instance, you can request multiple magnification levels of the same DICOM series, where each magnification level is represented by a different instance_uids:

"dicom_path": {
  "series_path": "https://dicomweb-store.uri/studies/1.2.3.4.5.6.7.8.9/series/1.2.3.4.5.6.7.8.10",
  "instance_uids": ["1.2.3.4.5.6.7.8.11", "1.2.3.4.5.6.7.8.12"],
}

Keep in mind that the total number of embeddings that you can request in one API call is capped by the service to a fixed limit that is set using _MAX_PATCHES_PER_REQUEST.

DICOM requirements

To ensure compatibility with the service, digital pathology Whole Slide Images (WSI) must be represented in DICOM format according to the requirements detailed in this section, which complement the HAI-DEF DICOM store requirements for the underlying storage system:

1. Supported Information Object Definitions (IODs): The service only supports the following IODs:

   Media Storage SOP Class UID / SOP Class UID	Name
   1.2.840.10008.5.1.4.1.1.77.1.6	VL Whole Slide Microscopy Image IOD
   1.2.840.10008.5.1.4.1.1.77.1.2	VL Microscopic Image IOD
   1.2.840.10008.5.1.4.1.1.77.1.3	VL Slide-coordinates Microscopic Image IOD

   The VL Whole Slide Microscopy Imaging is the most common microscopy DICOM IOD for storing and exchanging WSI, typically produced by high-resolution commercial slide scanners. These images are very large and organized as a multi-resolution pyramid within the DICOM file. This IOD can also include information like the slide's label, a thumbnail preview, and details about the scanned area.

   The VL Microscopic Image IOD and VL Slide-coordinates Microscopic Image IOD are used for storing images from microscopes with digital cameras. These images are not tiled. They are captured as a single, complete frame and represented by a single DICOM file.

2. Single pyramid representation: DICOM series that contain VL Whole Slide Microscopy Image IOD must contain one and only one representation of the image pyramid. DICOM imaging represented in either the VL Microscopic Image IOD or the VL Slide-coordinates Microscopic Image IOD must not be represented in the same series as VL Whole Slide Microscopy Image IOD imaging.

3. Uniform ICC color profile: If an ICC color profile is defined, the same ICC profile must be applicable to all levels of the image pyramid.

4. Single fragment encapsulated transfers In multi-frame images with encapsulated transfer syntax each frame's compressed data must be in one piece (one fragment). In single-frame images with encapsulated transfer syntax all the compressed data must be in one fragment.

5. Single focal planes: Images encoded with multiple focal planes are not supported.

6. Single optical path sequence: Images must be acquired using a single optical path and have one optical path sequence. Refer to Optical Path Module Attributes for details.

7. Transfer syntax transcoding: The service relies on the Transfer Syntax UID to transcode the image pixels. The following table lists the supported transcodes. If the service can't transcode the images itself, it falls back on the capabilities of the underlying DICOM storage system. If the DICOMs are from a Google Cloud DICOM store, this document on supported transfer syntaxes for transcoding applies.

   Supported UID	Name
   1.2.840.10008.1.2.4.50 (Recommended)	JPEG Baseline (Process 1): Default Transfer Syntax for Lossy JPEG 8-bit Image Compression
   1.2.840.10008.1.2.4.90	JPEG 2000 Image Compression (Lossless Only)
   1.2.840.10008.1.2.4.91	JPEG 2000 Image Compression
   1.2.840.10008.1.2.1	Uncompressed
   1.2.840.10008.1.2	Uncompressed
   '1.2.840.10008.1.2.1.99'	Deflated Explicit VR Little-Endian
   '1.2.840.10008.1.2.4.110'	JPEGXL_LOSSLESS (coming soon)
   '1.2.840.10008.1.2.4.111'	JPEGXL_JPEG (coming soon)
   1.2.840.10008.1.2.4.112'	JPEGXL (coming soon)

8. Required tags: The following tags are required:

   Tag	Name	Note
   (0002,0010)	TransferSyntraxUID
   (0008,0008)	ImageType
   (0008,0060)	Modality
   (0008,0016)	SOPClassUID
   (0008,0018)	SOPInstanceUID
   (0020,000E)	SeriesInstanceUID
   (0020,000D)	StudyInstanceUID
   (0020,9161)	ConcatenationUID	Required if instance is part of a concatenation (uncommon)
   (0020,9228)	ConcatenationFrameOffsetNumber	Required if instance is part of a concatenation (uncommon)
   (0020,9331)	DimensionalOrganizationType	Must be set to TILED_FULL for all multi-frame instances. Refer to Multi-frame Dimension Module Attributes for details.
   (0028,0002)	SamplesPerPixel	Must be set to 1 for monochrome or grayscale imaging and 3 for color imaging. Refer to Image Pixel Modul Attributes for details.
   (0028,0010)	Rows
   (0028,0011)	Columns
   (0028,0100)	BitsAllocated	Must be set to 8 indicating 8 bits per sample encoding. Refer to Image Pixel Module Attributes for details.
   (0028,0102)	HighBit	Must be set to 7 indicating little-endian encoding. Refer to Image Pixel Module Attributes for details.
   (0028,2000)	ICC Profile	Not a root level tag; expected in OpticalPathSequence (0048,0105) -> ICC_Profile (0028, 2000)
   (0028,9110)	PixelSpacing	Not a root level tag; expected in: Shared Functional Group Sequence (5200, 9229) -> Pixel Measures Sequence (0028, 9110) -> PixelSpacing (0028, 9110).
   (0048,0001)	ImagedVolumeWidth
   (0048,0002)	ImagedVolumeHeight
   (0048,0006)	TotalPixelMatrixColumns
   (0048,0007)	TotalPixelMatrixRows
   (7FE0,0010)	PixelData