Collaboration on Source Catalogue DM, ADQL and SkyNodes

[Clive Page]

On Wed, 21 Dec 2005, Jim Gray wrote:

> There is also the cross match that Tanu & Maria implemented in SQL.
> That may be the easiest way to do the implementation if the nodes all
> have a SQL backend.

I'd like to point out that if the data are already in a relational DBMS 
then by far the simplest way to do the cross-match, and in many cases also 
the fastest, is to use R-tree indexing and a spatial join.  I think the 
first astronomical use of this was by Andrea Baruffolo (see 
http://monet.ncsa.uiuc.edu/adass98/Proceedings/baruffoloa1/ ) but it has 
also been extensively tested here and documented on the AstroGrid wiki, 
see: http://wiki.astrogrid.org/bin/view/Astrogrid/DataDocs

Support for spatial indexing is now included in or readily available for 
DB2, Oracle, Informix, Sybase, MySQL, and Postgres, i.e. just about all 
the DBMS widely used in astronomy (with perhaps just one exception, which 
Jim can tell you about :-).

> But, getting objects into a node dominates all other costs (moving stuff
> thru xml is expensive).

Indeed that is a very serious problem.  I wonder if we can't solve this by 
using, instead of XML, some more efficient data format, e.g. one which 
holds tabular data in binary form with just the metadata in plain text. 
There's something called the "FITS table" with exactly these properties 
which perhaps astronomers should investigate :-)

Merry Christmas (translation for those in countries where Christianity is 
not the established religion: Happy Holidays).

-- 
Clive Page
Dept of Physics & Astronomy,
University of Leicester,
Leicester, LE1 7RH,  U.K.