<snip MIME signature>

Ian has agreed that binary metadata is probably the best thing. I
therefore propose a simple, extensible binary metadata format, primarily
aimed at implementing splitfiles in the not-too-distant future. It
provides limited extension capabilities in the areas of metadata,
splitfile codecs, document types, and ZIP manifests, and is reasonably
compact. It allows for various tricks which may be provided for in
future, such as DBR splitfiles, and piecing together different files in
a nonredundant splitfile. It allows for splitfiles of any conceivable
size, metadata of any conceivable size, ZIP manifests and ordinary
manifests. Limits will be imposed at the client level. Comments?

All numbers are in java big-endian.

8 bytes - magic number for freenet metadata
Wasted bytes, just being paranoid.

2 bytes - version number
0 for now.

1 byte - document type
 0  = SIMPLE_REDIRECT            - simple redirect (including splitfiles)
 1  = MULTI_LEVEL_METADATA       - multi-level metadata (fetch this key, then use it as metadata)
 2  = SIMPLE_MANIFEST            - ordinary manifest
 3  = ARCHIVE_MANIFEST           - ZIP/TAR manifest
 4  = ARCHIVE_INTERNAL_REDIRECT  - reserved for use in ZIP/TAR manifests, see below

If documentType == MULTI_LEVEL_METADATA:
 1 byte - number of levels (must decrease by 1 on each level!)
 1 byte - document type of final metadata
 8 bytes - length of final data

If documentType == SIMPLE_REDIRECT or SIMPLE_MANIFEST or ARCHIVE_MANIFEST or ARCHIVE_INTERNAL_REDIRECT
 2 bytes - flags
  bit 0 = FLAGS_SPLITFILE          - splitfile
  bit 1 = FLAGS_DBR                - DBR (splitfile + DBR *is* valid, not supported)
  bit 2 = FLAGS_NO_MIME            - no MIME type
  bit 3 = FLAGS_COMPRESSED_MIME    - compressed MIME type
  bit 4 = FLAGS_EXTRA_METADATA     - has extra metadata fields (ignored)
  bit 5 = FLAGS_FULL_KEYS          - redirects as full keys (invalid if splitfile)
  bit 6 = FLAGS_SPLIT_USE_LENGTHS  - reserved/unused
  bit 7 = FLAGS_COMPRESSED         - compressed splitfile

If documentType == ARCHIVE_MANIFEST:
  2 bytes - ARCHIVE_TYPE
   0 = ZIP ("application/zip", "application/x-zip") /* eventually get rid of ZIP support at some point */
   1 = TAR ("application/x-tar")

If flags |= FLAGS_SPLITFILE:
  8 bytes - real content length (compressed size, signed number)

If flags |= FLAGS_COMPRESSED:
  2 bytes - COMPRESSOR_TYPE
   0 = GZIP
   1 = BZIP2
   2 = LZMA
  8 bytes - decompressed content length (signed number)

If _NOT_ flags |= FLAGS_NO_MIME:
 If flags |= FLAGS_COMPRESSED_MIME:
  2 bytes - compressedMIMEValue (base MIME type IDs)
	See DefaultMIMETypes.java for the look up table

  If compressedMIMEValue & 0x7fff == 0x7fff:
	2 bytes - compressedMIMEParams
 else 
  1 byte  - length (N)
  N bytes - MIME Type (UTF-8 String)

If flags |= DBR: (not supported yet)
 4 bytes - period, in seconds
 4 bytes - offset, in seconds

If flags |= FLAGS_EXTRA_METADATA: (ignored)
 2 bytes - number of extra metadata fields
  For each extra metadata field:
   2 bytes - metadata field type
   1 byte  - length
   N bytes - metadata field specific information


If (!(flags |= FLAGS_SPLITFILE) && ((documentType == SIMPLE_REDIRECT) || (documentType == ARCHIVE_MANIFEST)):
 If FLAGS_FULL_KEYS 
  1 byte  - length of binary key
  N bytes - binary key (this is just a compressed FreenetURI, see FreenetURI.java)
 Else 
  5+32+32 bytes - raw binary form of a CHK

For a splitfile redirect:
2 bytes - algorithm ID
0 = no redundancy. Invalid unless bit 6 or 5 above is set.
1 = standard onion FEC algorithm
...
4 bytes - number of bytes of parameters
N bytes - parameters (e.g. number of segments etc)
4 bytes - number of blocks ( 2 bytes would have the 2GB limit )
4 bytes - number of check blocks

Followed by all the keys involved in the above format.


Multi-level metadata follows the above format, except there are no extra
fields. Multi-level metadata is likely to be primarily used for very
large splitfiles where the metadata does not fit into a single block.
A ZIP manifest is exactly the same - the above with no extra fields. The
file pointed to will contain a file called .metadata, which contains the
real metadata in manifest format.

Manifests:

4 bytes - number of redirects
1 byte - length of redirect name
N bytes - redirect name
4 bytes - length of sub-document

Then follows a document formatted as described above.
If the manifest is inside a ZIP manifest, then type 4 = redirect to
file in a ZIP manifest is valid:
1 byte - length of name
N bytes - name in ZIP file
