Sources provide access to source images. A source translates an identifier in a request URI to a source image locator, such as a pathname, in the particular type of underlying storage it is written to interface with. After checking whether the underlying object exists and is accessible, it can then provide access to it to other application components in a generalized way, so that the rest of the application does not need to know where an image resides, be it on the filesystem, a remote web server, or wherever.
Sources may provide access as a stream or a file. All sources can provide stream access, but only FilesystemSource can provide file access. This distinction is important because not all processors can read from streams. See Image Considerations for background on why this is an issue.
In a typical configuration, one source will handle all requests. It's also possible to select a source dynamically depending on the image identifier.
When the source.static configuration key is set to the name of a source, that source will handle all requests.
When a static source is not flexible enough, it is also possible to serve images from different sources from the same application instance. For example, you may have some images stored on a filesystem, and others stored in an S3 bucket. If you can differentiate their sources based on their identifier in code—either by analyzing the identifier string, or performing some kind of service request—you can use the delegate script mechanism to write a method to tell the image server which source to use for a given request.
To enable dynamic source selection, set the source.delegate configuration key to true, and implement the source() delegate method. For example:
class CustomDelegate
def source(options = {})
identifier = context['identifier']
# Here, you would perform some kind of analysis on `identifier`:
# parse it, look it up in a web service or database...
# and then return the name of the source to use to serve it.
'FilesystemSource'
end
endSee the Delegate Script section for general information about the delegate script.
I want to serve images located…
| On a filesystem… | …and the identifiers I use in URLs will correspond predictably to filesystem paths | FilesystemSource with BasicLookupStrategy |
| …and filesystem paths will need to be looked up (in a SQL database, search server, index file, etc.) based on their identifier | FilesystemSource with ScriptLookupStrategy | |
| On a local or remote web server… | …and the identifiers I use in URLs will correspond predictably to URL paths | HttpSource with BasicLookupStrategy |
| …and URL paths will need to be looked up (in a SQL database, search server, index file, etc.) based on their identifier | HttpSource with ScriptLookupStrategy | |
| In S3… | …and the identifiers I use in URLs will correspond predictably to object keys | S3Source with BasicLookupStrategy |
| …and object keys will need to be looked up (in a SQL database, search server, index file, etc.) based on their identifier | S3Source with ScriptLookupStrategy | |
| In Azure Storage | AzureStorageSource | |
| As binaries or BLOBs in a SQL database | JdbcSource | |
FilesystemSource maps a URL identifier to a filesystem path, for retrieving images on a local or mounted filesystem. This is the most efficient and compatible source and may be the only option for serving very large source images.
Two distinct lookup strategies are supported, defined by the FilesystemSource.lookup_strategy configuration option.
BasicLookupStrategy locates images by concatenating an identifier with a pre-defined path prefix and/or suffix. For example, with the following configuration options set:
# Note trailing slash!
FilesystemSource.BasicLookupStrategy.path_prefix = /usr/local/images/
FilesystemSource.BasicLookupStrategy.path_suffix =An identifier of image.jpg in the URL will resolve to /usr/local/images/image.jpg.
It's also possible to include a partial path in the identifier using URL-encoded slashes (%2F) as path separators. subdirectory%2Fimage.jpg in the URL would then resolve to /usr/local/images/subdirectory/image.jpg.
If you are operating behind a reverse proxy that is not capable of passing encoded URL characters through without decoding them, see the slash_substitute configuration key.
To prevent arbitrary directory traversal, BasicLookupStrategy will recursively strip out ../, /.., ..\, and \.. from identifiers before resolving the path.
FilesystemSource.BasicLookupStrategy.path_prefix is set as deep as possible. The shallower the path, the more of the filesystem that will be exposed.
Sometimes, BasicLookupStrategy will not offer enough control. Perhaps you want to serve images from multiple filesystems, or perhaps your identifiers are opaque and you need to perform a database or web service request to locate the corresponding images. With this lookup strategy, you can tell FilesystemSource to invoke a delegate method and capture the pathname it returns.
The delegate method, filesystemsource_pathname(), returns a pathname if available, or nil if not. (See the Delegate Script section for general information about the delegate script.) Examples follow:
require 'java'
java_import 'org.postgresql.Driver'
java_import 'java.sql.DriverManager'
class CustomDelegate
JDBC_URL = 'jdbc:postgresql://localhost:5432/mydatabase'
JDBC_USER = 'myuser'
JDBC_PASSWORD = 'mypassword'
# By making the connection static, we can avoid reconnecting every time
# the method is called, which would be expensive.
# See: https://docs.oracle.com/javase/8/docs/api/java/sql/DriverManager.html
@@conn = DriverManager.get_connection(JDBC_URL, JDBC_USER, JDBC_PASSWORD)
def filesystemsource_pathname(options = {})
identifier = context['identifier']
begin
# Note the use of prepared statements, which are safer than
# string concatenation.
sql = 'SELECT pathname FROM images WHERE identifier = ? LIMIT 1'
stmt = @@conn.prepare_statement(sql)
stmt.set_string(1, identifier)
results = stmt.execute_query
results.next
pathname = results.getString(1)
return pathname.present? ? pathname : nil
ensure
stmt&.close
end
end
endNote that several common Ruby database libraries (like the mysql and pgsql gems) use native extensions. These won't work in JRuby. Instead, the course of action above is to use the JDBC API via the JRuby-Java bridge. For this to work, a JDBC driver for your database must be installed on the Java classpath, and referenced in a java_import statement.
This very simple imaginary web service will return a pathname in the response body if an image was found, and an empty response body if not.
require 'net/http'
require 'cgi'
class CustomDelegate
def filesystemsource_pathname(options = {})
identifier = context['identifier']
uri = 'http://example.org/webservice/' + CGI.escape(identifier)
uri = URI.parse(uri)
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
return nil if response.code.to_i >= 400
response.body.present? ? response.body.strip : nil
end
endLike all sources, FilesystemSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:
HttpSource maps a URL identifier to an HTTP or HTTPS resource, for retrieving images from a web server. It uses a Jetty HTTP client internally.
HttpSource supports two distinct lookup strategies, defined by the HttpSource.lookup_strategy configuration option.
BasicLookupStrategy locates images by concatenating an identifier with a pre-defined URL prefix and/or suffix. For example, with the following configuration options set:
# Note trailing slash!
HttpSource.BasicLookupStrategy.url_prefix = http://example.org/images/
HttpSource.BasicLookupStrategy.url_suffix =An identifier of image.jpg in the URL will resolve to http://example.org/images/image.jpg.
It's possible to include a partial path in the identifier using URL-encoded slashes (%2F) as path separators. subpath%2Fimage.jpg in the URL would then resolve to http://example.org/images/subpath/image.jpg.
It's also possible to use a full URL as an identifier by leaving both of the above keys blank. In that case, an identifier of http%3A%2F%2Fexample.org%2Fimages%2Fimage.jpg in the URL will resolve to http://example.org/images/image.jpg.
If you are operating behind a reverse proxy that is not capable of passing encoded URL characters through without decoding them, see the slash_substitute configuration key.
Sometimes, BasicLookupStrategy will not offer enough control. Perhaps you want to serve images from multiple URLs, or perhaps your identifiers are opaque and you need to run a database or web service request to locate them. With this lookup strategy, you can tell HttpSource to invoke the httpsource_resource_info() delegate method and capture the URL it returns.
See the Delegate Script section for general information about the delegate script, and the FilesystemSource ScriptLookupStrategy section for examples of similar methods.
HTTP Basic authentication is supported.
HttpSource.BasicLookupStrategy.auth.basic.username and HttpSource.BasicLookupStrategy.auth.basic.secret configuration keys.Like all sources, HttpSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:
GET request is sent containing a Range header specifying a small range of data from the beginning of the resource.
Content-Type header, and its value is specific enough (i.e. not application/octet-stream), a format will be inferred from that.JdbcSource maps a URL identifier to a BLOB field, for retrieving images from a relational database. It does not require a custom schema and can adapt to any schema. The drawback is that some delegate methods must be implemented in order to obtain the information needed to run the SQL queries.
The application does not include any JDBC drivers, so a driver JAR for the desired database must be obtained separately and saved somewhere on the classpath.
The JDBC connection is initialized by the JdbcSource.url, JdbcSource.user, and JdbcSource.password configuration options. If the user or password are not necessary, they can be left blank. The connection string must use your driver's JDBC syntax:
jdbc:postgresql://localhost:5432/my_database
jdbc:mysql://localhost:3306/my_database
jdbc:microsoft:sqlserver://example.org:1433;DatabaseName=MY_DATABASEConsult the driver's documentation for details.
Then, the source needs to be told:
This method takes in an unencoded URL identifier and returns the corresponding database value of the identifier.
class CustomDelegate
def jdbcsource_database_identifier(options = {})
# If URL identifiers map directly to values in the database, simply
# return the identifier from the request context. Otherwise, you
# could transform it, perform a service request to look it up, etc.
context['identifier']
end
endThis method returns a media (MIME) type corresponding to the value returned by the jdbcsource_database_identifier() method. If the media type is stored in the database, this example will return an SQL statement to retrieve it.
class CustomDelegate
def jdbcsource_media_type(options = {})
'SELECT media_type ' +
'FROM some_table ' +
'WHERE some_identifier = ?'
end
endThis method may return nil; see Format Inference.
This method returns an SQL statement that selects the BLOB value corresponding to the value returned by the jdbcsource_database_identifier() method.
class CustomDelegate
def jdbcsource_lookup_sql(options = {})
'SELECT image_blob_column '
'FROM some_table '
'WHERE some_identifier = ?'
end
endLike all sources, JdbcSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:
S3Source maps a URL identifier to an Amazon Simple Storage Service (S3) object, for retrieving images from S3. S3Source can work with both AWS and non-AWS S3 endpoints.
S3Source can obtain its credentials from several different sources. It will first consult the S3Source.access_key_id and S3Source.secret_key keys in the application configuration. If those are not set, it will fall back to the strategy employed by the AWS SDK's DefaultAWSCredentialsProviderChain; see its class documentation for details.
BasicLookupStrategy locates images by concatenating an identifier with a pre-defined path prefix and/or suffix. For example, with the following configuration options set:
# Note trailing slash!
S3Source.BasicLookupStrategy.path_prefix = path/prefix/
S3Source.BasicLookupStrategy.path_suffix =An identifier of image.jpg in the URL will resolve to path/prefix/image.jpg within the bucket.
It's also possible to include a partial path in the identifier using URL-encoded slashes (%2F) as path separators. subpath%2Fimage.jpg in the URL would then resolve to path/prefix/subpath/image.jpg.
If you are operating behind a reverse proxy that is not capable of passing encoded URL characters through without decoding them, see the slash_substitute configuration key.
When your URL identifiers don't match your S3 object keys, ScriptLookupStrategy is available to tell S3Source to capture the object key returned by a method in your delegate script. The s3source_object_info() method returns a hash containing bucket and key keys, if an object is available, or nil if not. See the Delegate Script section for general information about the delegate script, and the FilesystemSource ScriptLookupStrategy section for examples of similar methods.
Like all sources, S3Source needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:
GET request will be sent with a Range header specifying a small range of data from the beginning of the resource.
Content-Type header is present in the response, and is specific enough (i.e. not application/octet-stream), a format will be inferred from that.AzureStorageSource maps a URL identifier to a Microsoft Azure Storage blob, for retrieving images from Azure Storage. It can be configured with the following options:
AzureStorageSource.account_nameAzureStorageSource.account_keyAzureStorageSource.container_nameAzureStorageSource.lookup_strategyAs of version 4.0, this source also supports shared access signatures as an alternative to hard-coding the account name, account key, and container name.
BasicLookupStrategy locates images by passing the URL identifier as-is to Azure Storage, with no additional configuration possible.
When your URL identifiers don't match your blob keys, ScriptLookupStrategy is available to tell AzureStorageSource to capture the blob key returned by a method in your delegate script.
The delegate method, azurestoragesource_blob_key(), returns a blob key string if available, or nil if not. See the Delegate Script section for general information about the delegate script, and the FilesystemSource ScriptLookupStrategy section for examples of similar methods.
Like all sources, AzureStorageSource needs to be able to figure out the format of a source image before it can be served. It uses the following strategy to do this:
HEAD request will be sent. If a Content-Type header is present in the response, and is specific enough (i.e. not application/octet-stream), a format will be inferred from that.GET request will be sent with a Range header specifying a small range of data from the beginning of the resource, and a format will be inferred from the magic bytes in the response body.