Sources

• Features
• Selection
  • Static
  • Dynamic
  • Which Source Should I Use?
• Implementations
  • FilesystemSource
  • HttpSource
  • S3Source
  • AzureStorageSource
  • JdbcSource

Sources provide access to source images, translating request URI identifiers into a source image locators, such as pathnames, in a particular type of underlying storage. After verifying that an underlying object exists and is accessible, a source can provide access to it to other application components in a generalized way.

Features

All sources can provide access to streams from which to read a resource, but only FilesystemSource can provide access to files. This distinction is important because not all processors can read from streams.

Of the sources that provide stream accesss, not all support random access. Random access can be of enormous benefit when using certain processor/source format combinations that can fully exploit it, which are currently:

• KakaduNativeProcessor with JPEG2000
• Java2dProcessor or JaiProcessor with pyramidal TIFF

* Using chunking

Selection

In a simple configuration, one source supplies all requests. But it's also possible to select a source dynamically depending on the image identifier.

Static Source

When the source.static configuration key is set to the name of a source, that source will supply all requests.

Dynamic Sources

When a static source is not flexible enough, it is also possible to serve images from different sources. For example, you may have some images stored on a filesystem, and others stored in an S3 bucket. If you can differentiate their sources based on their identifier in code—either by analyzing the identifier string, or performing some kind of service request—you can implement a delegate method to tell the image server from which source it should obtain the image.

To enable dynamic source selection, set the source.delegate configuration key to true, and implement the source() delegate method. For example:

class CustomDelegate
  def source(options = {})
    identifier = context['identifier']
    # Here, you would perform some kind of analysis on `identifier`:
    # parse it, look it up in a web service or database...
    # and then return the name of the source to supply it.
    'FilesystemSource'
  end
end

Which Source Should I Use?

I want to serve images located…

Implementations

FilesystemSource

FilesystemSource maps a URL identifier to a filesystem path. This is the most compatible source, and usually the most efficient as well.

Lookup Strategies

Two distinct lookup strategies are supported, defined by the FilesystemSource.lookup_strategy configuration option.

BasicLookupStrategy

BasicLookupStrategy locates images by concatenating an identifier with a pre-defined path prefix and/or suffix. For example, with the following configuration options set:

# Note trailing slash!
FilesystemSource.BasicLookupStrategy.path_prefix = /usr/local/images/
FilesystemSource.BasicLookupStrategy.path_suffix =

An identifier of image.jpg in the URL will resolve to /usr/local/images/image.jpg.

It's also possible to include a partial path in the identifier using URL-encoded slashes (%2F) as path separators. subdirectory%2Fimage.jpg in the URL would then resolve to /usr/local/images/subdirectory/image.jpg.

If you are operating behind a reverse proxy that is not capable of passing encoded URL characters through without decoding them, see the slash_substitute configuration key.

To prevent arbitrary directory traversal, BasicLookupStrategy will recursively strip out ../, /.., ..\, and \.. from identifiers before resolving the path.

ScriptLookupStrategy

Sometimes, BasicLookupStrategy will not offer enough control. Perhaps you want to serve images from multiple filesystems, or perhaps your identifiers are opaque and you need to perform a database or web service request to locate the corresponding images. With this lookup strategy, you can tell FilesystemSource to invoke a delegate method and capture the pathname it returns.

The delegate method, filesystemsource_pathname(), should return a pathname if available, or nil if not. Examples follow:

Example 1: Query a PostgreSQL database to find the pathname corresponding to a given identifier

require 'java'

java_import 'org.postgresql.Driver'
java_import 'java.sql.DriverManager'

class CustomDelegate

  JDBC_URL = 'jdbc:postgresql://localhost:5432/mydatabase'
  JDBC_USER = 'myuser'
  JDBC_PASSWORD = 'mypassword'

  # By making the connection static, we can avoid reconnecting every time
  # the method is called, which would be expensive.
  # See: https://docs.oracle.com/en/java/javase/11/docs/api/java.sql/java/sql/DriverManager.html
  @@conn = DriverManager.get_connection(JDBC_URL, JDBC_USER, JDBC_PASSWORD)

  def filesystemsource_pathname(options = {})
    identifier = context['identifier']
    begin
      # Note the use of prepared statements, which are safer than
      # string concatenation.
      sql = 'SELECT pathname FROM images WHERE identifier = ? LIMIT 1'
      stmt = @@conn.prepare_statement(sql)
      stmt.set_string(1, identifier)
      results = stmt.execute_query
      results.next
      pathname = results.getString(1)
      return pathname.present? ? pathname : nil
    ensure
      stmt&.close
    end
  end

end

Note that several common Ruby database libraries (like the mysql and pgsql gems) use native extensions. These won't work in JRuby. Instead, the course of action above is to use the JDBC API via the JRuby-Java bridge. For this to work, a JDBC driver for your database must be available on the Java classpath, and referenced in a java_import statement.

Example 2: Query a web service to find the pathname corresponding to a given identifier

This very simple mock web service returns a pathname in the response body when an image exists, and an empty response body if not.

require 'net/http'
require 'cgi'

class CustomDelegate
  def filesystemsource_pathname(options = {})
    identifier = context['identifier']
    uri = 'http://example.org/webservice/' + CGI.escape(identifier)
    uri = URI.parse(uri)

    http = Net::HTTP.new(uri.host, uri.port)
    request = Net::HTTP::Get.new(uri.request_uri)
    response = http.request(request)
    return nil if response.code.to_i >= 400

    response.body.present? ? response.body.strip : nil
  end
end

Format Inference

Like all sources, FilesystemSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:

1. If the file's filename contains an extension, the format is inferred from that.
2. If unsuccessful, and the identifier contains a filename extension, the format is inferred from that.
3. If unsuccessful, an attempt is made to infer a format from the file's "magic bytes."

HttpSource

HttpSource maps a URL identifier to an HTTP or HTTPS resource, for retrieving images from a web server. It uses an OkHttp client internally.

Lookup Strategies

HttpSource supports two distinct lookup strategies, defined by the HttpSource.lookup_strategy configuration option.

BasicLookupStrategy

BasicLookupStrategy locates images by concatenating an identifier with a pre-defined URL prefix and/or suffix. For example, with the following configuration options set:

# Note trailing slash!
HttpSource.BasicLookupStrategy.url_prefix = http://example.org/images/
HttpSource.BasicLookupStrategy.url_suffix =

An identifier of image.jpg in the URL will resolve to http://example.org/images/image.jpg.

A partial path can be included in the identifier by URL-encoding the path separator slashes (%2F). subpath%2Fimage.jpg in the URL would then resolve to http://example.org/images/subpath/image.jpg.

It's also possible to use a full URL as an identifier by leaving both of the above keys blank. In that case, an identifier of http%3A%2F%2Fexample.org%2Fimages%2Fimage.jpg in the URL will resolve to http://example.org/images/image.jpg.

If you are operating behind a reverse proxy that is not capable of passing encoded URL characters through without decoding them, see the slash_substitute configuration key.

ScriptLookupStrategy

Sometimes, BasicLookupStrategy will not offer enough control. Perhaps you want to serve images from multiple URLs, or perhaps your identifiers are opaque and you need to run a database or web service request to locate them. With this lookup strategy, you can tell HttpSource to invoke the httpsource_resource_info() delegate method and capture the request info (URL and optionally authentication credentials and/or request headers) it returns.

See the FilesystemSource ScriptLookupStrategy section for examples of similar methods.

Authentication

HTTP Basic authentication is supported.

• When using BasicLookupStrategy, auth info is set globally in the HttpSource.BasicLookupStrategy.auth.basic.username and HttpSource.BasicLookupStrategy.auth.basic.secret configuration keys.
• When using ScriptLookupStrategy, auth info can be returned from the delegate method.

Format Inference

Like all sources, HttpSource needs to be able to figure out the format of a source image before it can be served. It uses the strategy below to do this.

1. If the path component of the URI that HttpSource is trying to access contains an extension, the format is inferred from that.
2. If unsuccessful, and the identifier has a filename extension, the format is inferred from that.
3. If unsuccessful, and if the HEAD response contains a Content-Type header with a recognized value that is specific enough (not application/octet-stream, for example), a format is inferred from that.
4. If unsuccessful, and if the HEAD response contains an Accept-Ranges: bytes header, a GET request is sent containing a Range header specifying a small range of data from the beginning of the resource, and a format is inferred from the magic bytes in the response entity.
5. If still unsuccessful, the request fails.

Random Access

Since version 4.1, this source supports random access by requesting small chunks of image data as needed, as opposed to all of it. This may improve efficiency—possibly massively—when reading small portions of large images in certain formats (see below). Conversely, it may reduce efficiency when reading large portions of images.

In order for this technique to work:

1. The HttpSource.chunking.enabled configuration key must be set to true;
2. The HTTP server must support the Range header, as advertised by the presence of an Accept-Ranges: bytes header in a HEAD response;
3. One of the following processor/source format combinations must be used:
   • KakaduNativeProcessor with JPEG2000
   • Java2dProcessor or JaiProcessor with multi-resolution TIFF

Note that when chunking is in effect, the processor.stream_retrieval_strategy configuration key is ignored, effectively behaving as if it were set to StreamStrategy. (See Retrieval Strategies.) Chunking is meant to obviate the expense of the other strategies.

The HTTP client library used by this source changed in version 5.0. The old client (Jetty) used $JAVA_HOME/jre/lib/security/cacerts as its trust store, whereas the new client (OkHttp) uses the value of the javax.net.ssl.trustStore VM argument. After migrating from 4.1, if you encounter errors like, "PKIX path building failed," try setting the value of that VM argument to the aforementioned path, or the path of some other trust store.

S3Source

S3Source maps a URL identifier to a Simple Storage Service (S3) object. S3Source can work with both AWS and non-AWS S3 endpoints.

Credentials Sources

Credentials are obtained from the following sources in order of priority:

1. The aws.accessKeyId and aws.secretKey system properties
2. The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
3. The S3Source.access_key_id and S3Source.secret_key keys in the application configuration
4. The ~/.aws/credentials file
5. The Amazon EC2 container service (if the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set and the security manager has permission to access it)
6. Instance profile credentials delivered through the Amazon EC2 metadata service

Lookup Strategies

BasicLookupStrategy

BasicLookupStrategy locates images by concatenating an identifier with a pre-defined path prefix and/or suffix. For example, with the following configuration options set:

# Note trailing slash!
S3Source.BasicLookupStrategy.path_prefix = path/prefix/
S3Source.BasicLookupStrategy.path_suffix =

An identifier of image.jpg in the URL will resolve to path/prefix/image.jpg within the bucket.

It's also possible to include a partial path in the identifier using URL-encoded slashes (%2F) as path separators. subpath%2Fimage.jpg in the URL would then resolve to path/prefix/subpath/image.jpg.

If you are operating behind a reverse proxy that is not capable of passing encoded URL characters through without decoding them, see the slash_substitute configuration key.

ScriptLookupStrategy

When your URL identifiers don't match your S3 object keys, ScriptLookupStrategy is available to tell S3Source to capture the object key returned by a method in your delegate class. The s3source_object_info() method should return a hash containing bucket and key keys, if an object is available, or nil if not. See the FilesystemSource ScriptLookupStrategy section for examples of similar methods.

Format Inference

Like all sources, S3Source needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:

1. If the object key has a recognized filename extension, the format is inferred from that.
2. Otherwise, if the source image's URI identifier has a recognized filename extension, the format is inferred from that.
3. Otherwise, a GET request is sent with a Range header specifying a small range of data from the beginning of the resource.
   1. If a Content-Type header is present in the response, and is specific enough (i.e. not application/octet-stream), a format is inferred from that.
   2. Otherwise, a format is inferred from the magic bytes in the response body.

Random Access

This source supports random access since version 4.1. See the HttpSource section for an explanation of this feature.

AzureStorageSource

AzureStorageSource maps a URL identifier to a Microsoft Azure Blob Storage blob.

Lookup Strategies

BasicLookupStrategy

BasicLookupStrategy locates images by passing the URL identifier as-is to Azure Storage, with no additional configuration possible.

ScriptLookupStrategy

When your URL identifiers don't match your blob keys, ScriptLookupStrategy is available to tell AzureStorageSource to capture the blob key returned by a method in your delegate class.

The delegate method, azurestoragesource_blob_key(), should return a blob key string if available, or nil if not. See the FilesystemSource ScriptLookupStrategy section for examples of similar methods.

Format Inference

Like all sources, AzureStorageSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:

1. If the blob key has a recognized filename extension, the format is inferred from that.
2. Otherwise, if the source image's URI identifier has a recognized filename extension, the format is inferred from that.
3. Otherwise, a HEAD request is sent. If a Content-Type header is present in the response, and is specific enough (i.e. not application/octet-stream), a format is inferred from that.
4. Otherwise, a GET request is sent with a Range header specifying a small range of data from the beginning of the resource, and a format is inferred from the magic bytes in the response body.

Random Access

This source supports random access since version 4.1. See the HttpSource section for an explanation of this feature.

JdbcSource

JdbcSource maps a URL identifier to a BLOB field in a relational database. It does not require a custom schema and can adapt to any schema, but some delegate methods must be implemented in order to obtain the information needed to run the SQL queries.

The application does not include any JDBC drivers, so a driver JAR for the desired database must be obtained separately and saved somewhere on the classpath.

The JDBC connection is initialized by the JdbcSource.url, JdbcSource.user, and JdbcSource.password configuration options. If the user or password are not necessary, they can be left blank. The connection string must use your driver's JDBC syntax:

jdbc:postgresql://localhost:5432/my_database
jdbc:mysql://localhost:3306/my_database
jdbc:microsoft:sqlserver://example.org:1433;DatabaseName=MY_DATABASE

Consult the driver's documentation for details.

Then, the source needs to be told:

1. The database value corresponding to a given identifier
2. The media type corresponding to that value
3. The SQL statement that retrieves the BLOB value corresponding to that value

Database Identifier Retrieval Method

This method takes in an unencoded URL identifier and returns the corresponding database value of the identifier.

class CustomDelegate
  def jdbcsource_database_identifier(options = {})
    # If URL identifiers map directly to values in the database, simply
    # return the identifier from the request context. Otherwise, you
    # could transform it, perform a service request to look it up, etc.
    context['identifier']
  end
end

Media Type Retrieval Method

This method should return a media (MIME) type corresponding to the value returned by the jdbcsource_database_identifier() method. If the media type is stored in the database, this example will return an SQL statement to retrieve it.

class CustomDelegate
  def jdbcsource_media_type(options = {})
    'SELECT media_type ' +
        'FROM some_table ' +
        'WHERE some_identifier = ?'
  end
end

This method may return nil; see Format Inference.

BLOB Retrieval SQL Method

This method should return an SQL statement that selects the BLOB value corresponding to the value returned by the jdbcsource_database_identifier() method.

class CustomDelegate
  def jdbcsource_lookup_sql(options = {})
    'SELECT image_blob_column '
        'FROM some_table '
        'WHERE some_identifier = ?'
  end
end

Format Inference

Like all sources, JdbcSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:

1. If the media type retrieval method returns either a recognized media type, or an SQL query that can be invoked to obtain a recognized media type, the corresponding format is used.
2. If the source image's URI identifier has a recognized filename extension, the format is inferred from that.
3. Otherwise, the blob retrieval SQL is executed to obtain a small range of data from the beginning of the resource, and an attempt is made to infer a format from its magic bytes.