From mjg at cacr.caltech.edu Mon Oct 10 13:09:10 2011 From: mjg at cacr.caltech.edu (Matthew Graham) Date: Mon, 10 Oct 2011 13:09:10 -0700 Subject: VOSpace 2.0 issues Message-ID: <3B2F21ED-BD96-425D-917E-656B3637BE06@cacr.caltech.edu> Hi, Pat raised a number of questions about VOSpace 2.0 in a number of emails to this list over the summer and I've finally found time to look at them and respond: (1) Synchronous transfers Section 3.8 lists the various REST bindings. The binding for the transfer details for synchronous jobs should be: /{transfers}/{job-id}/results/transferDetails There should also be an endpoint for POST synchronous jobs: /{sync} which redirects to the /{transfers}/{job-id}/results/transferDetails endpoint in response. The idea behind synchronous transfers was to offer convenience methods specifically for transferring data in/out of a VOSpace via HTTP PUT and GET. VOSpace is normally protocol agnostic but these are so common that they warranted special provision. The semantics of transferring data into a VOSpace mean that the operation has to be a HTTP POST to a job endpoint whereas transferring data from a VOSpace is facilitated by an additional parameter on the HTTP GET URL (view=data). This is an unfortunate asymmetry but people did not want to do job transfer negotiation when retrieving data in the synchronous case since the protocol was already understood as HTTP. If you want to allow for multiple protocol choices or protocol switching in downloading data then this has to be negotiated through the asynchronous channel. (2) Third-party (server-server) transfers The client negotiates both ends of the transfer. We cannot assume that there will just be transfers between VOSpaces and it is an extra onus on implementors to make the VOSpace service negotiate when the transfer is to another VOSpace but not when it is not. Far easier to just get the client sort out the details in all cases and not just some. (3) Faults in deleteNode By definition, the parent of the node to be deleted should exist and be a ContainerNode. However, it is not directly specified as the node to be deleted but implied. If it does not exist or is not a ContainerNode then that reflects an internal inconsistency in the VOSpace service. A 500 is therefore a more appropriate error than a 404 in this context. Similar arguments apply to the LinkNode as parent. (4) URIs in move/copy For the sake of convenience, it was decided to model move and copy (within the same space) as asynchronous transfers. The same (transfer) syntax could then be used across the board. The inconvenience is that one has to use vos URIs. URIs from different authorities should throw an exception. Cheers, Matthew From dave.morris at bristol.ac.uk Tue Oct 11 04:33:44 2011 From: dave.morris at bristol.ac.uk (Dave Morris) Date: Tue, 11 Oct 2011 12:33:44 +0100 Subject: copy/move/pushFrom/pullTo In-Reply-To: <201109201318.35028.patrick.dowler@nrc-cnrc.gc.ca> References: <201109201318.35028.patrick.dowler@nrc-cnrc.gc.ca> Message-ID: <4E942998.50706@bristol.ac.uk> Hi all, On 20/09/11 21:18, Patrick Dowler wrote: > > Can the client simply specify the destination as a (vos) URI? eg let the > server negotiate the protocols? If not, is there some scenario that justifies > forcing the client to do this? One reason for requiring the client to handle the initial transfer protocol negotiation is that VOSpace isn't just one specification. We have two versions already, and may have more in the future. If the servers were required to negotiate the transfer protocol between themselves based on just a vos:// URI, then it would require all services to be able to support all versions of the protocol. New services would have to support all the previous versions, and more problematic, would be updating all the old services to implement all the new versions. If we want to transfer from an old 1.x SOAP based server to a new 2.x REST based server, then as long as our client can handle both versions of the protocol it can negotiate a common data transfer protocol between the two services. The services don't need to support the same versions of the VOSpace protocol. Having said that, there is a way to ask the services to do the negotiation themselves, which involves treating the VOSpace protocol as a transfer protocol. As part of the registry capabilities registration we have a standard URI for each version of the VOSpace protocol. If a service implementation included a client capable of handling transfer negotiations using a particular version of the VOSpace protocol, then the service could use the standard URI to advertise that in the list of transfer protocols it offers. If both the services offer 'VOSpace 1.x' or 'VOSpace 2.x' as one of the protocol options for a transfer, then the client can initiate the transfer using that as the transfer protocol, effectively asking the services to use the common version of the VOSpace protocol to negotiate the details of the transfer between themselves. This leaves it up to the service provider to decide which versions of the VOSpace protocol their service handles. For example, a new REST based v2.x service does not have to implement the full v1.x SOAP API, but it may include a v1.x SOAP client library that enables it to negotiate transfers with an older v1.x SOAP service. In which case the new service would be registered in the registry as a REST based v2.x service, but it could offer the v1.x SOAP protocol as one of the options for a transfer. I don't know if anyone one has provided this level of protocol negotiation in a live service, but it is available as part of both the v1.x SOAP and 2.x REST based service specifications. Hope this helps, Dave From patrick.dowler at nrc-cnrc.gc.ca Tue Oct 11 13:19:39 2011 From: patrick.dowler at nrc-cnrc.gc.ca (Patrick Dowler) Date: Tue, 11 Oct 2011 13:19:39 -0700 Subject: VOSpace 2.0 issues In-Reply-To: <3B2F21ED-BD96-425D-917E-656B3637BE06@cacr.caltech.edu> References: <3B2F21ED-BD96-425D-917E-656B3637BE06@cacr.caltech.edu> Message-ID: <201110111319.39483.patrick.dowler@nrc-cnrc.gc.ca> Thanks for the clarifications. For transfers, we now have pullFrom, pushTo, and move implemented and should get copy, pushFrom, and pullTo implemented over the next week or so. I will have to see how to finesse the /{sync} resource to work with the otherwise async UWS /{transfers} resource. However, I still don't like this explanation of the 500 error in delete (and other ops): On 2011-10-10 13:09:10 Matthew Graham wrote: > (3) Faults in deleteNode > > By definition, the parent of the node to be deleted should exist and be a > ContainerNode. However, it is not directly specified as the node to be > deleted but implied. If it does not exist or is not a ContainerNode then > that reflects an internal inconsistency in the VOSpace service. A 500 is > therefore a more appropriate error than a 404 in this context. Similar > arguments apply to the LinkNode as parent. Sure, if I am making a DELETE /nodes/path/to/something then clearly "path" and "to" must exist. In VOSpace they can be containers or links. However, if the client requests this and the resource does not exist, it should still be a 404. It is additionally helpful to say ContainerNotFound when "path" or "to" are not found, and NodeNotFound when "something" is not found, but it is still a plain old 404 in http land. There is no inconsistent server state that warrants a 500 - it is just a user trying to act on a resource that doesn't exist. Or, are you saying that my service can return a 404 NodeNotFound if /path/to/something does not exist independent of how far one traverses the path before not finding a node? That would be fine too, but then I don't see how one ever gets the 500 ContainerNotFound. As for one of them being a LinkNode, the spec can make this operation illegal if it likes, but should it be a "bad request" (400 LinkFound)? possibly confused, -- Patrick Dowler Tel/T?l: (250) 363-0044 Canadian Astronomy Data Centre National Research Council Canada 5071 West Saanich Road Victoria, BC V9E 2M7 Centre canadien de donnees astronomiques Conseil national de recherches Canada 5071, chemin West Saanich Victoria (C.-B.) V9E 2M7 From andre.schaaff at astro.unistra.fr Wed Oct 12 08:38:05 2011 From: andre.schaaff at astro.unistra.fr (Andre Schaaff) Date: Wed, 12 Oct 2011 17:38:05 +0200 Subject: IVOA Grid and Web Services sessions In-Reply-To: <4E931845.7090608@astro.unistra.fr> References: <4E931845.7090608@astro.unistra.fr> Message-ID: <4E95B45D.1010908@astro.unistra.fr> Hello, It seems that my mail has not been diffused on the grid list last monday. regards Andr? -------- Original Message -------- Subject: IVOA Grid and Web Services sessions Date: Mon, 10 Oct 2011 18:07:33 +0200 From: Andre Schaaff Reply-To: andre.schaaff at astro.unistra.fr Organization: CDS To: grid at ivoa.net CC: Andreas Wicenec Hello, If you are attending the Pune interop meeting and wish to contribute to the GWS sessions, please let me know the title and the time you need. We have 2 sessions called VOSpace and UWS, but it is completely open. So feel free to propose a presentation outside these two topics. Andre -- Andre Schaaff Observatoire astronomique 11, rue de l'Universite F-67000 Strasbourg