Edward.J.Shaya.1 at gsfc.nasa.gov
Thu Jan 22 13:27:33 PST 2004
Guy Rixon wrote:
>Ed Shaya wrote:
>>OK. Now the crux of this all is that the Tabular form of this
>>is done by a auto-flattener that simply turns this into a
>>series of tables exactly as an XML to DB converter does.
>>Now you have a compact form to transfer and one that can go
>>right into your local/private DB. If it is an object
>>relational DB you should be able to access the data with the
>>original true XML.
>Interesting. I have a few questions:
>Where do we get the "auto-flattener"? Is it some generic tool, or does IVOA
>have to develop it?
At the metadata WG I was asked that and I said look at XMLDB. But, I
now realize that the
problem is really quite easy solved by simple transformation (of the
XSLT type or an equivalent).
Our knowledge is describable by the following simple serialization of a
<!ELEMENT Objects (Object+)> <!-- Root of Objects.dtd -->
<!ELEMENT Object (SimpleQuantity*,QuantitywithMetadata*,
type CDATA> <!-- specify type of Object (star, galaxy,
table, datacenter) -->
<!-- Just a property of the object with a value, error, and units -->
<!ELEMENT SimpleQuantity (Value,Error, Units)>
<!ATTLIST SimpleQuantity name CDATA>
<! --Like the simple one but add some metadata in a hierarchy -->
<!ELEMENT QuantitywithMetadata (Value, Error, Units, Metadata)
<!ATTLIST QuantitywithMetadata name CDATA>
<!ELEMENT Metadata (Quantity+)>
<!ELEMENT ArrayQuantity (image*, spectra*, timeSeries*) >
<!-- We'll just hold these in FITS for now, but of course we could
construct separate Quantity tables for them -->
<!ELEMENT image (FITSFile)>
<!ELEMENT FITSFile EMPTY>
<!ATTLIST FITSFile href URL>
<!ELEMENT spectra (FITSFile)>
<!ELEMENT timeSeries (FITSFile)>
<!ELEMENT image EMPTY>
<!ATTLIST image href URL>
<!-- Now we have a Quantity which holds memberOf, as in galaxies are
members of clusters -->
<!ELEMENT QuantityWithObjects (Object*)>
<!ATTLIST type CDATA>
Done, more or less. I am thinking this is actually too general, and
the VO.xsd will be a restriction of this.
Now we want to flatten this.
At the top level, take all Objects of the same type and put them into a
do the same for all other types.
Be sure to create a master table that has two columns (ObjectType,
Put Object/@name into first column.
Each SimpleQuantity goes into three columns or so, value, error, units.
If all the units are the same, then can collapse that column into
Do the same with QuantityWithMetadata but now an additional column is
created and the Metadata is placed there whole as either an xs:anytype
or as a string.
The ArrayQuantity goes into a <Group type="images"> and the columns are
"type of file": FITS in this case, and "URL": give the href value.
Finally QuantityWithObjects becomes a link
If there are several different type of Objects in this node, then it may
need to be broken
out into separate tables.
>How about the "auto-un-flattener" for recovering the hierarchical form from
>the tabular form? Is it always lossless? Does it need artifacts other than
>the results tables ex the auto-flattener to work?
The crumbler is created by exactly reversing the proceedure.
>Is the "compact form" some existing standard or do we have to define it?
I think VOTable can handle this. I am not sure about the links down the
hierarchy of tables.
>Is it in one logical chunk of meany (e.g. one per table)?
>Is it streamable?
Either VOTable or this Objects.xml are streamable, I think.
>How is it entered into the various RDBMS we use? (One would expect to need
>adaptors for each DBMS.)
The Table view of the data should fly into a DBMS. And hopefully, you
can extract it with
the above Objects.dtd or its equivalent xsd.
>Can we make an auto-flattener that produces existing formats like binary FITS?
Could, if FITS could handle a hierarchy of tables. Or, if you allow one
FITSTable to hyperlink to another.
>Guy Rixon gtr at ast.cam.ac.uk
>Institute of Astronomy Tel: +44-1223-337542
>Madingley Road, Cambridge, UK, CB3 0HA Fax: +44-1223-337523
More information about the dm