VOTable alternative? Mapping Rich Data to DBMSs
cgp at star.le.ac.uk
Tue Jan 27 02:54:15 PST 2004
On Tue, 27 Jan 2004, Martin Hill wrote:
> In fact few (if any?) of our databases are flat. However the results of
> our queries usually return a 'joined' table that is, so we would need to
> unflatten it to recreate structure. This is a way of re-introducing the
> natural structure of the information - with the normal query result we
> get, for example, shape cells for objects that don't have shape.
I'm not sure I agree with that, or maybe we disagree on the meaning of the
word "flat" (or the word "is" like Bill Clinton did).
A FITS table is a good way of representing a table, a huge improvement on
the way that the relational DBMS does this, as it includes a lot of
important metadata. These are metadata in both senses, structural
metadata such as the number of columns and data type, and astronomical
metadata such details of the telescope which produced the dataset and the
reference frame for the coordinates.
FITS has a rather crude way of representing metadata, essentially only
what you can get on an 80-column punched-card, but even though its still
way ahead of any commercial DBMS. The columnar metadata isn't represented
like this in FITS, but actually it is best set out as another table, for
example like this:
1 RA real*8 degrees F8.4 not null
2 DECL real*8 degrees F8.4 not null
3 VMAG real*4 magnitudes F5.2 nullable
4 XPOS int*2 pixels I4 nullable
and so on. (of course even FITS can handle more metadata than this, such
as scale factors, and a text description, but this is just an example).
So a tabular dataset could be represented quite well as two tables (one of
actual data, one of columnar metadata) plus some other overall metadata
which applies to the whole table (name of observer, equinox, epoch, etc).
The last are mostly simple and scalar (but not quite all, suppose you want
seeing conditions as a function of time?) - would VOTable cope with this?
This isn't quite a flat structure, but pretty nearly. Even if you join
one such table with another, you don't make the resulting structure any
more or less flat than the input structures (which is where I fail to
understand Martin's remarks).
As far as the columnar metadata goes:
* FITS stores them but not as a table and all mixed up with other
* VOTable puts them in an XML structure the tabular nature of which isn't
at all evident;
* an RDBMS holds a tiny bit of columnar metadata in internal tables which
you can't get at easily, and refuses to store anything else (such as
None of these solutions is very good; they mean that we have no obvious
way of passing columnar metadata to the Registry, for example, except by
inventing yet another format for them (which existing tools like TOPCAN
then cannot handle).
On the whole, however, I think I agree with Guy and Norman: the case for
VOTable2 is not yet proven.
Dept of Physics & Astronomy,
University of Leicester, Tel +44 116 252 3551
Leicester, LE1 7RH, U.K. Fax +44 116 252 3311
More information about the votable