Binary data (branch from future of VOTable)

[Tony Linde]

> I have yet to see a standard for bulk binary data transport 
> that is significantly better than FITS.  Hence the wrapping 

There's a BinX format floating about from (I think) the grid world - and Guy
told me about some DFDL (pronounced 'daffodil' apparently) which is a way of
allowing xml-based tools to read binary format data. I don't know much more
about it.

Cheers,
Tony. 

> -----Original Message-----
> From: owner-votable at eso.org [mailto:owner-votable at eso.org] On 
> Behalf Of Arnold Rots
> Sent: 14 April 2004 14:34
> To: VOTable mailing list
> Subject: Re: Comments on V1.1 - Future of VOTable
> 
> Now that this side of the Atlantic has woken up and surveyed
> (bleary-eyed) the wisdom that has been dispensed from the 
> other side while we were blissfully asleep, we might a well 
> join in :-)
> 
> I'll repeat my old mantra (slightly disagreeing with what Clive said
> earlier): FITS defined the syntax of the metadata but failed 
> to define the semantics - and that's turned out to have been 
> a considerable problem.  HEASARC/OGIP tried to fix that with 
> a set of conventions, but such efforts were very much limited 
> to sub-communities: high energy, optical, and radio each have 
> their own set of idiosyncratic conventions; and that's 
> troublesome for a VO.
> 
> To expand on the reversibility that Mark quoted:
> So, defining an XML-based metadata standard that is purposely 
> semantic in nature is a very sensible thing to do and XML is 
> well-suited for that (better than FITS).  On the other hand, 
> I have yet to see a standard for bulk binary data transport 
> that is significantly better than FITS.  Hence the wrapping 
> concept - which I think is actually a very sensible, but also 
> accpetable, way of transporting data and information.  
> Extract the metadata from the FITS headers (requiring 
> software that understands the peculiar dialect of a 
> particular FITS file), put it in a universally understandable 
> XML document (for the sake of argument, let's say a VOTable), 
> and send it with the FITS file containing the data - that 
> seems like a perfectly acceptable and practical solution (at 
> least to me). 
> 
> And, of course, if you can translate a FITS dialect to a 
> VOTable, you can try to do the reverse and feed the new file 
> into your existing FITS-based analysis package - or you can 
> use something more modern.
> 
> In summary:
> - The data transport mechanism of FITS ain't broken -> don't fix it.
> - The metadata semantics in FITS is a problem -> replace it 
> by something better (VOTable or whatever).
> 
> Anyway, that's the assumption I have been working from.
> Cheers,
> 
>   - Arnold
> 
> Mark Taylor wrote:
> > On Wed, 14 Apr 2004, martin hill wrote:
> > 
> > > While I appreciate that VOTable can be used to wrap FITS, 
> this is (I 
> > > hope) a temporary measure while we sort out our data models and 
> > > representations.  We should recognise it as such and put future 
> > > effort into producing suitable solutions, not in 
> overloading VOTable 
> > > to represent all data formats.  FITS files too have 
> limited structures.
> > 
> > I don't see it as a temporary measure.  VOTable is not supposed to 
> > represent all data formats, FITS is a special case.  From 
> sec 2.3 of 
> > the VOTable document:
> > 
> >    "the transformation of FITS to VOTable is meant to be reversible"
> > 
> > The kind of data (though not metadata) which can be held in 
> a VOTable 
> > is by design compatible with the FITS binary table format, so that 
> > requiring the possible storage of VOTable bulk data using a FITS 
> > binary table does not introduce any additional limitations (put 
> > another way, VOTable explicitly accepts the limitations of 
> FITS binary 
> > tables as far as pure data storage streams go).
> > 
> > Mark
> > 
> > -- 
> > Mark Taylor    Starlink Programmer     Physics,  Bristol 
> University, UK
> > m.b.taylor at bris.ac.uk +44-117-928-8776 
> > http://www.star.bris.ac.uk/~mbt/
> > 
> --------------------------------------------------------------
> ------------
> Arnold H. Rots                                Chandra X-ray 
> Science Center
> Smithsonian Astrophysical Observatory                tel:  +1 
> 617 496 7701
> 60 Garden Street, MS 67                              fax:  +1 
> 617 495 7356
> Cambridge, MA 02138                             
> arots at head.cfa.harvard.edu
> USA                                     
> http://hea-www.harvard.edu/~arots/
> --------------------------------------------------------------
> ------------
>