Changes to VOSpace specification

[Dave Morris]

Paul Harrison wrote:

> I guess that I am arguing for what we could do if there was no  
> ListStatusNodes() method and this was done with the existing API - we  
> could use this part of the URL for multiple things - I was not really  
> very careful with the syntax but if we had something like
>
> vos://org.test!vospace/container/node?operation=transfer_status
>
> we could then do things like
>
> vos://org.test!vospace/container/node? 
> operation=get_view&view=jpeg_cutout

We are designing a SOAP webservice - so why add yet another syntax for 
calling methods.
If we didn't already have a SOAP service, then you might have a case.

Just add status nodes and ListStatusNodes() to version-1.1 of the 
schema, and we can have it up and running by the next IVOA meeting.

All of the operations can be handled in the existing VOSpace 
specification using Java Servlets or their equivalent
integrated within the VOSpace service. We don't _need_ callbacks in 
version-1.0.

> which leads me to the thought that perhaps we are being too purist  
> not having a direct getContent(URI) SOAP call that does return the  
> "content" of the URI as a synchronous result, in-line. With the  
> formal XML being a choice of an "any" xml or a small xml structure  
> with a <data> element with base64 encoded data in it ....

How about this :

   <method uri="[dime-get-inline]"/>

Where the URI points to a resource that says something like this :

   ivo://net.ivoa.vospace/transfer-methods/dime-get-inline-1.0
       This is a VOSpace transfer method that uses the Direct Internet 
Message Encapsulation (DIME) message format to return the node contents 
as a SOAP attachment.
       Details of the DIME specification can be found at here 
[http://msdn.microsoft.com/library/en-us/dnglobspec/html/draft-nielsen-dime-02.txt] 


       Usage:
       The client calls PullFromVOSpace specifying this transfer-method 
in the request.
       If the request is valid, then the server will respond with the 
standard PullFromVOSpace,
       response, and attach the data node contents to the end of the 
message as a DIME attachment.

       The transfer completes when the SOAP response is received by the 
client.
       No additional steps are required.

       If the server is unable to attach the data node contents to the 
message, then it either
       will response with the appropriate error message, or a list of 
other transfer-methods
       if they are available.

Or this :

   <method uri="[mtom-get-inline]"/>

Where the URI points to a resource that says something like this :

   ivo://net.ivoa.vospace/transfer-methods/mtom-get-inline-1.0
       This is a VOSpace transfer method that uses the Message 
Transmission Optimization Mechanism (MTOM) message format to return the 
node contents as a SOAP attachment.
       Details of the MTOM specification can be found here 
[http://www.w3.org/TR/soap12-mtom/]

       Usage:
       The client calls PullFromVOSpace specifying this transfer-method 
in the request.
       If the request is valid, then the server will respond with the 
standard PullFromVOSpace,
       response, and attach the data node contents to the end of the 
message as a MTOM attachment.

       The transfer completes when the SOAP response is received by the 
client.
       No additional steps are required.

       If the server is unable to attach the data node contents to the 
message, then it either
       will response with the appropriate error message, or a list of 
other transfer-methods
       if they are available.

Both of which are possible within the current VOSpace version 1.0 
specification.

> ....  (actually pretty much the same as Amazon S3 - getObject method). 
> OK-  this is not as efficient as the multiprotocol data transfer 
> methods ....

See : http://solutions.amazonwebservices.com/connect/ann.jspa?annID=144
Posted : Oct 23, 2006 11:21 PM
/In a few weeks time, you will no longer be able to use the Amazon S3 
SOAP interface's PutObjectInline or GetObjectInline for objects that are 
larger than 1 megabyte in size. After November 21, affected requests 
will fail with the new error response InlineDataTooLargeError. To put or 
get objects that are larger than 1 megabyte in size, you will need to 
either use DIME attachments with SOAP requests or use our REST 
interface. We apologize for any inconvenience this change may cause you.

Please be assured that deprecating existing functionality is not a 
change we take lightly. We are making this change because transferring 
large objects inline in the SOAP message (as happens with 
PutObjectInline and GetObjectInline) has significant negative 
performance implications; it is a mechanism only suited to transferring 
small amounts of data. The SOAP-based alternative, using DIME 
attachments, now has wide support in SOAP toolkits, is more efficient 
and does not suffer from the same performance bottlenecks. It also 
allows us to optimize our service to give you better performance./

> .... but it does give the client something very simple that it can  
> implement ....

What is so difficult in a client implementing this :

   <method uri="[http-get]"/>

Where the URI points to a resource that says something like this :

   ivo://net.ivoa.vospace/transfer-methods/http-get
       This is a VOSpace transfer method that uses the standard http-get 
transport protocol.
       Details of the transport protocol are available here 
[http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3].

       Usage:
       The client calls PullFromVOSpace specifying this transfer-method 
in the request.
       If the request is valid, then the server will respond with the 
standard PullFromVOSpace,
       response, and add a <transfer-method> element containing the 
endpoint URL to fetch the data from.

       The client uses the URL to download the data using a standard 
http-get request.
       The transfer method completes automatically when the client has 
received the data,
       and no notification callbacks are required.

> .... and perhaps we can also say that servers are allowed to  throw a 
> "fetch with multiprotocol transfer method instead" exception  if they 
> feel that a data object is being requested is too large to  return 
> in-line ....

This would mean adding a new exception to the VOSpace service WSDL, one 
that isn't connected with the core VOSpace API but is specific to one 
transfer-method only.

Given that the current version-1.0 specification can support 
[dime-get-inline], [mtom-get-inline] and [http-get], I don't see the 
case for modifying the VOSpace specification to support a custom version 
of http-get, using ? to encode method params outside the SOAP message 
and a non standard way of encoding the data.

As yet, no show stopper bugs found - the current version-1.0 
specification is good enough.
Lets get that through the process and then we can add status nodes, 
ListStatusNodes() and a generic set of notification callbacks to the 
next version.

Dave