New SkyNode and ADQL - reply to many emails (esp. Tony)

[Wil O'Mullane]

New ADQL and SkyNode spec have been posted on twiki ..
http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOQL

Please note links to the Skynode WSDL and ADQL XSD are also there.

Tony & Clive had many questions and issues - I try to deal with each below. You may or may not wish to read all of it . 
You have been warned..


SKYNODE
+++++++++++
SINGLE SPEC- Too wide in scope ---
All agreed too much was in the original spec. This is now split into ADQL and SkyNode. Recently The Standard Interfaces Spec has been made available. So section 4 is replaced with a single requirement referencing the standard interfaces spec.

The portal part indeed should also be removed. The Architecture sections says enough and the Portal sections should be put in a new Portal spec coming latter. So Section 6 goes.

>What is the intention of the SkyNode interface? Is it to be the *only* way
>in which any VOQL query can be submitted? Does every data query service have
>to implement this interface? If not, what others are there and where do they
>fit in?

The intention for SkyNode Spec was as a standard way to accept and ADQL query (not VOQL) . Almost certainly it would not be the only way but it would be standard way. Many people already want to implement this interface but there is no standard so I don't think we should not do anything about it for now - it should move forward. This is meant to be a step on from Cone service - a little more useful.

>- is the basic/full distinction valid? What if a basic service can return a
>QueryCost: how is this specified? The functions that a query service can
>perform should be in the metadata; they should each be namespaced and
>versioned with their own specs (which might be part of this one).
I think this is valid - in another one of your mails you ask that we state what is the minimum a node needs to do to be IVOA - that is BASIC. The document goes on to describe other desirable features to enable portals to do more advanced queries. Any particular node code exist anywhere in the spectrum. I have added text to clarify this.

>5.2.1:
>QI-3:
>- we need a better way of signifying features than is extensible and does
>not require hard-coding of metadata in the registry xsd.
But this is what the metadata is for is it not. Granted though Xmatch and Region may be in the functions list but the accepts UCDS really has no other place unless it is done using a translation function like findColNameForUCD.
I happily remove the other two but think we need to keep AcceptsUCDS.

>QI-4/5/6...:
>- these ports need to be better specified or is this in the wsdl? What is
>the format of the return data? Xml? If so, we need a validating xsd.
Yes they are specified in the WSDL which was the intention. I shall post the WSDL alongside the next version of the DOC.


>- Tables, Columns, Formats and Functions are/should be part of the metadata
>and so do not need separate calls.
There well long discussions on this also at several meetings and the only solution I could see was to stick to the skyquery way of doing things. If you want this metadata in the registry you go back and get it from each call. The Resource Metadata only contains the description of the node and the base url.

>QI-8:
>- don't understand this. Is this saying that one query can return the data
>in multiple formats or just that a query can specify the format (*should be
>from a namespaced list*) it wants the data returned in? what do the abstract
>class and subclasses mean? Are these Java classes or XSD classes?
The formats are listed by a call to the Formats interface. I am assuming a string token- formats gives you a list of strings you give back one. You get your result in that one format. The subclasses are detailed in the WSDL.

Extra text added about QueryCost

>QI-12:
>- this seems to wrap several functions which are better split out: the
>ability to *create* a table and populate it (whether via VOTable or via some
>other ADQL query); the ability to Xmatch. If this function is saying that
>VOTable and some query result can be cross-matched without loading the the
>VOTable as a db table, how is this supposed to be done? Are the calculations
>provided the *only* way of doing XMatching? What if someone wants to provide
>the same funciton but with a better method?
This is the part which I had said needed more work in Decemeber and why I suggested perhaps delaying this spec going forward for recommendation. 
Theoretically yes you may do any Xmatch you like. How you do the Xmatch is your decision. We will load the table in a temporary space and then perform the Xmatch.
This is and Exec plan are purely for OpenSkyQuery. 

>QI-13:
>- what is the purpose of this function? What structure is the ExecPlan? Why
>do results have to come back as VOTable? There must be a more efficient way
>of passing data between DBMSs. Why do we need a portal url? All the 'nodes'
>should be other skynode services, shouldn't they? In which case, we only
>need their ResourceIDs to identify them?
>**surely all this is workflow functionality, rather than data query. If a
>query service can support cross-db querying then we just need the ADQL to
>reflect that**
As stated above this is ofr OpenSkyQuery - this is the way SkyQuery currently works and it has been bough in here also. Yes this is work flow. I am not sure I want to turn every node into a portal though. SkyQuery broke this apart nicely and this spec attempts to do the same. There is a minimum a node needs to do the portal does all the really nasty parts.

ADQL
++++++++
Reference added to region specification - no spec actually available yet.

>ADQL-2:
>- this illustrates my point. If the xsd IS the standard then we don't need
>to state this, if not, then all the other aspects of the ADQL also deserve
>explanation.
I am unsure a truly exhaustive document is useful here. There is a premise that we are based on SQL. The document tries to detail where we are tighter or differ from SQL . This is surly more useful than an exhaustive SQL description where inevitably subtleties would be lost.
Perhaps we need a new ADQL-1  ADQL shall be  based on SQL.
 

>ADQL-6 
Added more text to clarify version  and its meaning.

>The REGION spec is also important, and again only referenced by an xml.
>Google found several versions for me, it's not clear which is the master.
>The only reference here is an example:
 > Region('CIRCLE J2000 19.5 -36.7 0.02')
AH! There was a reference to HTMDLL which seems to have disappeared !
The current Parser only deals with Circle this will be remedied in the near future.

SKYQL. --
Tony  suggests dropping this term. I think this would be fine so in place of SKYQL we simply have ADQL string representation. 
ALL -> Is there any Consensus opinion on that ?? 

>ADQL-8/9/10:
>- this is confusing and suggests that SkyQL is different to ADQL. Why do we
>need all this description?
There was extensive discussion on how to include regions in the Text representation. These iterate the ways in which this may be done. I would not remove them without saying this in some other way.

UNITS
We had some discussion in NVO about this. We feel this may be added to ADQL later if needed. For now it will bring more problems than solutions. We feel we should continue with ADQL as is for the near term


Phew ...
wil