Handling of file:// URIs:
This directory contains code to map basic boto connection, bucket, and key
operations onto files in the local filesystem, in support of file://
URI operations.
Bucket storage operations cannot be mapped completely onto a file system
because of the different naming semantics in these types of systems: the
former have a flat name space of objects within each named bucket; the
latter have a hierarchical name space of files, and nothing corresponding to
the notion of a bucket. The mapping we selected was guided by the desire
to achieve meaningful semantics for a useful subset of operations that can
be implemented polymorphically across both types of systems. We considered
several possibilities for mapping path names to bucket + object name:
1) bucket = the file system root or local directory (for absolute vs
relative file:// URIs, respectively) and object = remainder of path.
We discarded this choice because the get_all_keys() method doesn't make
sense under this approach: Enumerating all files under the root or current
directory could include more than the caller intended. For example,
StorageUri("file:///usr/bin/X11/vim").get_all_keys() would enumerate all
files in the file system.
2) bucket is treated mostly as an anonymous placeholder, with the object
name holding the URI path (minus the "file://" part). Two sub-options,
for object enumeration (the get_all_keys() call):
a) disallow get_all_keys(). This isn't great, as then the caller must
know the URI type before deciding whether to make this call.
b) return the single key for which this "bucket" was defined.
Note that this option means the app cannot use this API for listing
contents of the file system. While that makes the API less generally
useful, it avoids the potentially dangerous/unintended consequences
noted in option (1) above.
We selected 2b, resulting in a class hierarchy where StorageUri is an abstract
class, with FileStorageUri and BucketStorageUri subclasses.
Some additional notes:
BucketStorageUri and FileStorageUri each implement these methods:
- clone_replace_name() creates a same-type URI with a
different object name - which is useful for various enumeration cases
(e.g., implementing wildcarding in a command line utility).
- names_container() determines if the given URI names a container for
multiple objects/files - i.e., a bucket or directory.
- names_singleton() determines if the given URI names an individual object
or file.
- is_file_uri() and is_cloud_uri() determine if the given URI is a
FileStorageUri or BucketStorageUri, respectively
