Collaboration on Source Catalogue DM, ADQL and SkyNodes -- "Spatial is special"
Jim.Gray at microsoft.com
Wed Dec 21 09:55:33 PST 2005
The paradox for me, being a database guy,
is that Rtrees seem to lose out to the zone algorithm for crossmatch
and seem to lose out to HTM for point-near-point or point-in-polygon.
And the footprint queries (polygon-overlaps-polygon) seems to want the
algebraic approach (halfspace intersections and then simplification).
One might think that Rtrees would beat HTM, but they seem to have
problems with the poles and the Cartesian representation of points
And Healpix and Igloo have their place in this "space".
As they say: "Spatial is special."
It seems to have a special access method for each application,
And every researcher seems to have a favorite access method or
From: owner-voql at eso.org [mailto:owner-voql at eso.org] On Behalf Of Clive
Sent: Wednesday, December 21, 2005 8:04 AM
To: voql at ivoa.net
Subject: RE: Collaboration on Source Catalogue DM, ADQL and SkyNodes
On Wed, 21 Dec 2005, Jim Gray wrote:
> There is also the cross match that Tanu & Maria implemented in SQL.
> That may be the easiest way to do the implementation if the nodes all
> have a SQL backend.
I'd like to point out that if the data are already in a relational DBMS
then by far the simplest way to do the cross-match, and in many cases
also the fastest, is to use R-tree indexing and a spatial join. I think
the first astronomical use of this was by Andrea Baruffolo (see
http://monet.ncsa.uiuc.edu/adass98/Proceedings/baruffoloa1/ ) but it has
also been extensively tested here and documented on the AstroGrid wiki,
Support for spatial indexing is now included in or readily available for
DB2, Oracle, Informix, Sybase, MySQL, and Postgres, i.e. just about all
the DBMS widely used in astronomy (with perhaps just one exception,
which Jim can tell you about :-).
> But, getting objects into a node dominates all other costs (moving
> stuff thru xml is expensive).
Indeed that is a very serious problem. I wonder if we can't solve this
by using, instead of XML, some more efficient data format, e.g. one
which holds tabular data in binary form with just the metadata in plain
There's something called the "FITS table" with exactly these properties
which perhaps astronomers should investigate :-)
Merry Christmas (translation for those in countries where Christianity
is not the established religion: Happy Holidays).
Dept of Physics & Astronomy,
University of Leicester,
Leicester, LE1 7RH, U.K.
More information about the voql