Collaboration on Source Catalogue DM, ADQL and SkyNodes
cgp at star.le.ac.uk
Wed Dec 21 08:04:24 PST 2005
On Wed, 21 Dec 2005, Jim Gray wrote:
> There is also the cross match that Tanu & Maria implemented in SQL.
> That may be the easiest way to do the implementation if the nodes all
> have a SQL backend.
I'd like to point out that if the data are already in a relational DBMS
then by far the simplest way to do the cross-match, and in many cases also
the fastest, is to use R-tree indexing and a spatial join. I think the
first astronomical use of this was by Andrea Baruffolo (see
http://monet.ncsa.uiuc.edu/adass98/Proceedings/baruffoloa1/ ) but it has
also been extensively tested here and documented on the AstroGrid wiki,
Support for spatial indexing is now included in or readily available for
DB2, Oracle, Informix, Sybase, MySQL, and Postgres, i.e. just about all
the DBMS widely used in astronomy (with perhaps just one exception, which
Jim can tell you about :-).
> But, getting objects into a node dominates all other costs (moving stuff
> thru xml is expensive).
Indeed that is a very serious problem. I wonder if we can't solve this by
using, instead of XML, some more efficient data format, e.g. one which
holds tabular data in binary form with just the metadata in plain text.
There's something called the "FITS table" with exactly these properties
which perhaps astronomers should investigate :-)
Merry Christmas (translation for those in countries where Christianity is
not the established religion: Happy Holidays).
Dept of Physics & Astronomy,
University of Leicester,
Leicester, LE1 7RH, U.K.
More information about the voql