From francois.bonnarel at astro.unistra.fr Thu Sep 1 02:08:20 2011 From: francois.bonnarel at astro.unistra.fr (Francois Bonnarel) Date: Thu, 01 Sep 2011 11:08:20 +0200 Subject: utype for STC region in SIAP query response In-Reply-To: <20110829184203.GA4578@ari.uni-heidelberg.de> References: <4E5B63BC.6030009@astro.unistra.fr> <4E5B8657.7010607@astro.unistra.fr> <4E5B8996.2070709@astro.unistra.fr> <20110829184203.GA4578@ari.uni-heidelberg.de> Message-ID: <4E5F4B84.9070905@astro.unistra.fr> Hi Markus, all, Markus Demleitner a ?crit : > Hi Thomas, hi List, > > [utype for region withing SIAP] > > On Mon, Aug 29, 2011 at 02:44:06PM +0200, Thomas Boch wrote: > >> I wish this is something that could be standardized in a forecoming >> revision of the SIAP document, so that clients can easily find out >> which FIELD support the STC-S description. >> > Well, I'd say: the utype should reflect whatever place the region has > within the the "Simple Image" data model. Since there's no such > thing as far as I know, there should be no utype at all (though a UCD > might come in handy).. > This is not totally true actually and has been discussed during first discussions around SIA2. The draft posted in Novembre 2009 (by the Garching IVOA meeting) has utypes proposals for images (and cubes) which are actually nothing different from the Observation data model as it was seen at that time... Apart from the mapping (wcs ) part and a few other details most of it is also valid for generic datasets or observation descriptions... In the mean time the Observation data Model Core components have been reworked within the frame of ObsTap. There is an ObsCore utype for the kind of stuff Thomas is looking for, as I Told allready... What should be done now is to review the 2009 SIA draft and make the utypes totally consistent with Obscore > However, we do have an STC data model, and there's a Note [1] on how > to embed the mapping of data in VOTables to it [disclaimer: mainly > written by yours truly]. So, what I'd suggest is to just use that. > If I may say so, it's much easier in implementation than one might > think, and it'll remain the way it is even if we decide to define a > Simple Image data model. > > In the note, there's already an example for embedding a region. I > think a future SIAP spec would profit from recommending a full STC > record, and I'll be happy to figure one out -- just not right now > since I'm on vacation. > I think this is an option as long as you use the correct obs/sia2 utype on one of the Fieldref/GROUPS... But as it is not that consistent with SIA/Obstap query response structure (which don't say anything about defining GROUPS in the response VOTABLE) a field with xtype : STC-S and utype "sia:Char.SpatialAxis.Coverage.Support.Area" Best Regards Fran?ois > Cheers, > > Markus > > [1] http://www.ivoa.net/Documents/Notes/VOTableSTC/ > > -- ===================================================================== Fran?ois Bonnarel Observatoire Astronomique de Strasbourg CDS (Centre de donn?es 11, rue de l'Universit? astronomiques de Strasbourg) F--67000 Strasbourg (France) Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr --------------------------------------------------------------------- From francois.bonnarel at astro.unistra.fr Thu Sep 1 02:16:35 2011 From: francois.bonnarel at astro.unistra.fr (Francois Bonnarel) Date: Thu, 01 Sep 2011 11:16:35 +0200 Subject: utype for STC region in SIAP query response In-Reply-To: <4E5CB292.1090307@astro.unistra.fr> References: <4E5B63BC.6030009@astro.unistra.fr> <4E5B8657.7010607@astro.unistra.fr> <4E5B8996.2070709@astro.unistra.fr> <4E5CB292.1090307@astro.unistra.fr> Message-ID: <4E5F4D73.50904@astro.unistra.fr> Hi Thomas, To complete my answer to Markus we could have something like I think something like this can be proposed (if not yet the case) in SIA2... this will be consistent with ObsTap... and could also be included in the next version of SSA et al. For current SIA services, maybe we could have a short recommendation or note to propose using the same as an additional field in the SIA1 Query response Cheers Fran?ois > To illustrate my request, here is is how the STC-S FIELD is described > in the query response of three different SIAP services : > > Service 1 : xtype="adql:REGION" datatype="char" arraysize="*"> > > Service 2 : ucd="phys.area;obs.field" unit="deg"> > > Service 3 : arraysize="*"> > > 3 different services, 3 different ways to express the same thing. > Having a unique unambiguous way to detect the STC-S field would be > very helpful for client consuming these services. > Cheers, > > Thomas > > Thomas Boch wrote: >> Fran?ois, >> >> I wish this is something that could be standardized in a forecoming >> revision of the SIAP document, so that clients can easily find out >> which FIELD support the STC-S description. >> >> Thomas >> >> Fran?ois Bonnarel wrote: >>> Hi Thomas, >>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area >>> for this . We have something similar in the SIA2 prototypes >>> I guess eventually for sia we will have something like >>> sia:Char.SpatialAxis.Coverage.Support.Area >>> although the problem of the name space is still to be discussed... >>> By the way you can write this field as an STC-S feature ... >>> Cheers >>> Fran?ois >>> Le 29/08/2011 12:02, Thomas Boch a ?crit : >>>> Hi, >>>> >>>> Embedding a STC region in the SIAP query response is becoming more >>>> and more widespread, and we would like to support this feature in >>>> Aladin. >>>> I would like to know if there is a standard (or de facto standard) >>>> utype to characterize the FIELD holding the STC region description. >>>> Cheers, >>>> >>>> Thomas >>>> >>> >> >> > > > To illustrate my request, here is is how the STC-S FIELD is described > in the query response of three different SIAP services : > > Service 1 : xtype="adql:REGION" datatype="char" arraysize="*"> > > Service 2 : ucd="phys.area;obs.field" unit="deg"> > > Service 3 : arraysize="*"> > > 3 different services, 3 different ways to express the same thing. > Having a unique unambiguous way to detect the STC-S field would be > very helpful for client consuming these services. > Cheers, > > Thomas > > Thomas Boch wrote: >> Fran?ois, >> >> I wish this is something that could be standardized in a forecoming >> revision of the SIAP document, so that clients can easily find out >> which FIELD support the STC-S description. >> >> Thomas >> >> Fran?ois Bonnarel wrote: >>> Hi Thomas, >>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area >>> for this . We have something similar in the SIA2 prototypes >>> I guess eventually for sia we will have something like >>> sia:Char.SpatialAxis.Coverage.Support.Area >>> although the problem of the name space is still to be discussed... >>> By the way you can write this field as an STC-S feature ... >>> Cheers >>> Fran?ois >>> Le 29/08/2011 12:02, Thomas Boch a ?crit : >>>> Hi, >>>> >>>> Embedding a STC region in the SIAP query response is becoming more >>>> and more widespread, and we would like to support this feature in >>>> Aladin. >>>> I would like to know if there is a standard (or de facto standard) >>>> utype to characterize the FIELD holding the STC region description. >>>> Cheers, >>>> >>>> Thomas >>>> >>> >> >> > > Thomas Boch a ?crit : > To illustrate my request, here is is how the STC-S FIELD is described > in the query response of three different SIAP services : > > Service 1 : xtype="adql:REGION" datatype="char" arraysize="*"> > > Service 2 : ucd="phys.area;obs.field" unit="deg"> > > Service 3 : arraysize="*"> > > 3 different services, 3 different ways to express the same thing. > Having a unique unambiguous way to detect the STC-S field would be > very helpful for client consuming these services. > Cheers, > > Thomas > > Thomas Boch wrote: >> Fran?ois, >> >> I wish this is something that could be standardized in a forecoming >> revision of the SIAP document, so that clients can easily find out >> which FIELD support the STC-S description. >> >> Thomas >> >> Fran?ois Bonnarel wrote: >>> Hi Thomas, >>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area >>> for this . We have something similar in the SIA2 prototypes >>> I guess eventually for sia we will have something like >>> sia:Char.SpatialAxis.Coverage.Support.Area >>> although the problem of the name space is still to be discussed... >>> By the way you can write this field as an STC-S feature ... >>> Cheers >>> Fran?ois >>> Le 29/08/2011 12:02, Thomas Boch a ?crit : >>>> Hi, >>>> >>>> Embedding a STC region in the SIAP query response is becoming more >>>> and more widespread, and we would like to support this feature in >>>> Aladin. >>>> I would like to know if there is a standard (or de facto standard) >>>> utype to characterize the FIELD holding the STC region description. >>>> Cheers, >>>> >>>> Thomas >>>> >>> >> >> > > -- ===================================================================== Fran?ois Bonnarel Observatoire Astronomique de Strasbourg CDS (Centre de donn?es 11, rue de l'Universit? astronomiques de Strasbourg) F--67000 Strasbourg (France) Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr --------------------------------------------------------------------- From arots at head.cfa.harvard.edu Thu Sep 1 11:04:41 2011 From: arots at head.cfa.harvard.edu (Arnold Rots) Date: Thu, 1 Sep 2011 14:04:41 -0400 (EDT) Subject: utype for STC region in SIAP query response In-Reply-To: <4E5F4D73.50904@astro.unistra.fr> Message-ID: <201109011804.p81I4f9Q025872@xebec.cfa.harvard.edu> Here is what, in my view, is the correct utype for a footprint region: stc:ObservationLocation.AstroCoordArea.Region Pretty simple; though one might quibble whether it should be: stc:ObsDataLocation.ObservationLocation.AstroCoordArea.Region - Arnold Francois Bonnarel wrote: [ Charset ISO-8859-1 unsupported, converting... ] > Hi Thomas, > To complete my answer to Markus > we could have something like > > arraysize="*" utype="sia:Char.SpatialAxis.Coverage.Support.Area" > > > I think something like this can be proposed (if not yet the case) > in SIA2... > this will be consistent with ObsTap... and could also be included > in the next version of SSA et al. > > For current SIA services, maybe we could have a short > recommendation or note to propose using > the same as an additional field in the SIA1 Query response > Cheers > Fran?ois > > To illustrate my request, here is is how the STC-S FIELD is described > > in the query response of three different SIAP services : > > > > Service 1 : > xtype="adql:REGION" datatype="char" arraysize="*"> > > > > Service 2 : > ucd="phys.area;obs.field" unit="deg"> > > > > Service 3 : > arraysize="*"> > > > > 3 different services, 3 different ways to express the same thing. > > Having a unique unambiguous way to detect the STC-S field would be > > very helpful for client consuming these services. > > Cheers, > > > > Thomas > > > > Thomas Boch wrote: > >> Fran?ois, > >> > >> I wish this is something that could be standardized in a forecoming > >> revision of the SIAP document, so that clients can easily find out > >> which FIELD support the STC-S description. > >> > >> Thomas > >> > >> Fran?ois Bonnarel wrote: > >>> Hi Thomas, > >>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area > >>> for this . We have something similar in the SIA2 prototypes > >>> I guess eventually for sia we will have something like > >>> sia:Char.SpatialAxis.Coverage.Support.Area > >>> although the problem of the name space is still to be discussed... > >>> By the way you can write this field as an STC-S feature ... > >>> Cheers > >>> Fran?ois > >>> Le 29/08/2011 12:02, Thomas Boch a ?crit : > >>>> Hi, > >>>> > >>>> Embedding a STC region in the SIAP query response is becoming more > >>>> and more widespread, and we would like to support this feature in > >>>> Aladin. > >>>> I would like to know if there is a standard (or de facto standard) > >>>> utype to characterize the FIELD holding the STC region description. > >>>> Cheers, > >>>> > >>>> Thomas > >>>> > >>> > >> > >> > > > > > > To illustrate my request, here is is how the STC-S FIELD is described > > in the query response of three different SIAP services : > > > > Service 1 : > xtype="adql:REGION" datatype="char" arraysize="*"> > > > > Service 2 : > ucd="phys.area;obs.field" unit="deg"> > > > > Service 3 : > arraysize="*"> > > > > 3 different services, 3 different ways to express the same thing. > > Having a unique unambiguous way to detect the STC-S field would be > > very helpful for client consuming these services. > > Cheers, > > > > Thomas > > > > Thomas Boch wrote: > >> Fran?ois, > >> > >> I wish this is something that could be standardized in a forecoming > >> revision of the SIAP document, so that clients can easily find out > >> which FIELD support the STC-S description. > >> > >> Thomas > >> > >> Fran?ois Bonnarel wrote: > >>> Hi Thomas, > >>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area > >>> for this . We have something similar in the SIA2 prototypes > >>> I guess eventually for sia we will have something like > >>> sia:Char.SpatialAxis.Coverage.Support.Area > >>> although the problem of the name space is still to be discussed... > >>> By the way you can write this field as an STC-S feature ... > >>> Cheers > >>> Fran?ois > >>> Le 29/08/2011 12:02, Thomas Boch a ?crit : > >>>> Hi, > >>>> > >>>> Embedding a STC region in the SIAP query response is becoming more > >>>> and more widespread, and we would like to support this feature in > >>>> Aladin. > >>>> I would like to know if there is a standard (or de facto standard) > >>>> utype to characterize the FIELD holding the STC region description. > >>>> Cheers, > >>>> > >>>> Thomas > >>>> > >>> > >> > >> > > > > > Thomas Boch a ?crit : > > To illustrate my request, here is is how the STC-S FIELD is described > > in the query response of three different SIAP services : > > > > Service 1 : > xtype="adql:REGION" datatype="char" arraysize="*"> > > > > Service 2 : > ucd="phys.area;obs.field" unit="deg"> > > > > Service 3 : > arraysize="*"> > > > > 3 different services, 3 different ways to express the same thing. > > Having a unique unambiguous way to detect the STC-S field would be > > very helpful for client consuming these services. > > Cheers, > > > > Thomas > > > > Thomas Boch wrote: > >> Fran?ois, > >> > >> I wish this is something that could be standardized in a forecoming > >> revision of the SIAP document, so that clients can easily find out > >> which FIELD support the STC-S description. > >> > >> Thomas > >> > >> Fran?ois Bonnarel wrote: > >>> Hi Thomas, > >>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area > >>> for this . We have something similar in the SIA2 prototypes > >>> I guess eventually for sia we will have something like > >>> sia:Char.SpatialAxis.Coverage.Support.Area > >>> although the problem of the name space is still to be discussed... > >>> By the way you can write this field as an STC-S feature ... > >>> Cheers > >>> Fran?ois > >>> Le 29/08/2011 12:02, Thomas Boch a ?crit : > >>>> Hi, > >>>> > >>>> Embedding a STC region in the SIAP query response is becoming more > >>>> and more widespread, and we would like to support this feature in > >>>> Aladin. > >>>> I would like to know if there is a standard (or de facto standard) > >>>> utype to characterize the FIELD holding the STC region description. > >>>> Cheers, > >>>> > >>>> Thomas > >>>> > >>> > >> > >> > > > > > > > -- > ===================================================================== > Fran?ois Bonnarel Observatoire Astronomique de Strasbourg > CDS (Centre de donn?es 11, rue de l'Universit? > astronomiques de Strasbourg) F--67000 Strasbourg (France) > > Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html > Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr > --------------------------------------------------------------------- > -------------------------------------------------------------------------- Arnold H. Rots Chandra X-ray Science Center Smithsonian Astrophysical Observatory tel: +1 617 496 7701 60 Garden Street, MS 67 fax: +1 617 495 7356 Cambridge, MA 02138 arots at head.cfa.harvard.edu USA http://hea-www.harvard.edu/~arots/ -------------------------------------------------------------------------- From dtody at nrao.edu Thu Sep 1 16:40:03 2011 From: dtody at nrao.edu (Douglas Tody) Date: Thu, 1 Sep 2011 17:40:03 -0600 (MDT) Subject: utype for STC region in SIAP query response In-Reply-To: <201109011804.p81I4f9Q025872@xebec.cfa.harvard.edu> References: <201109011804.p81I4f9Q025872@xebec.cfa.harvard.edu> Message-ID: This would be a fine solution, considering only STC; however it is perfectly legal for a DM to be based upon other models (in a way defined by the new model such as Char or SIA) and define new, more application-specific Utypes. There is an especially strong argument for this for essential elements of the new model so that we have a well defined and self consistent model. "Falling through" to reference arbitrary elements of another model, with its own externally defined namespace, is fine for extension but could significantly expand the complexity of the new model as a client application might then need to fully understand multiple models to be able to deal with the data. In the case here, Char defines strong and consistent concepts for the measurement axes, Coverage, Support, etc. which are important to the Char model, and merely uses STC-S to represent the datum in question. Regarding UCDs for representing footprints in SIA-1, this can only be done by extension and there is no standard way to do it at present. We really need SIAV2 to address this sort of concern. - Doug On Thu, 1 Sep 2011, Arnold Rots wrote: > Here is what, in my view, is the correct utype for a footprint region: > > stc:ObservationLocation.AstroCoordArea.Region > > Pretty simple; though one might quibble whether it should be: > > stc:ObsDataLocation.ObservationLocation.AstroCoordArea.Region > > > - Arnold > > > Francois Bonnarel wrote: > [ Charset ISO-8859-1 unsupported, converting... ] >> Hi Thomas, >> To complete my answer to Markus >> we could have something like >> >> > arraysize="*" utype="sia:Char.SpatialAxis.Coverage.Support.Area" > >> >> I think something like this can be proposed (if not yet the case) >> in SIA2... >> this will be consistent with ObsTap... and could also be included >> in the next version of SSA et al. >> >> For current SIA services, maybe we could have a short >> recommendation or note to propose using >> the same as an additional field in the SIA1 Query response >> Cheers >> Fran?ois >>> To illustrate my request, here is is how the STC-S FIELD is described >>> in the query response of three different SIAP services : >>> >>> Service 1 : >> xtype="adql:REGION" datatype="char" arraysize="*"> >>> >>> Service 2 : >> ucd="phys.area;obs.field" unit="deg"> >>> >>> Service 3 : >> arraysize="*"> >>> >>> 3 different services, 3 different ways to express the same thing. >>> Having a unique unambiguous way to detect the STC-S field would be >>> very helpful for client consuming these services. >>> Cheers, >>> >>> Thomas >>> >>> Thomas Boch wrote: >>>> Fran?ois, >>>> >>>> I wish this is something that could be standardized in a forecoming >>>> revision of the SIAP document, so that clients can easily find out >>>> which FIELD support the STC-S description. >>>> >>>> Thomas >>>> >>>> Fran?ois Bonnarel wrote: >>>>> Hi Thomas, >>>>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area >>>>> for this . We have something similar in the SIA2 prototypes >>>>> I guess eventually for sia we will have something like >>>>> sia:Char.SpatialAxis.Coverage.Support.Area >>>>> although the problem of the name space is still to be discussed... >>>>> By the way you can write this field as an STC-S feature ... >>>>> Cheers >>>>> Fran?ois >>>>> Le 29/08/2011 12:02, Thomas Boch a ?crit : >>>>>> Hi, >>>>>> >>>>>> Embedding a STC region in the SIAP query response is becoming more >>>>>> and more widespread, and we would like to support this feature in >>>>>> Aladin. >>>>>> I would like to know if there is a standard (or de facto standard) >>>>>> utype to characterize the FIELD holding the STC region description. >>>>>> Cheers, >>>>>> >>>>>> Thomas >>>>>> >>>>> >>>> >>>> >>> >>> >>> To illustrate my request, here is is how the STC-S FIELD is described >>> in the query response of three different SIAP services : >>> >>> Service 1 : >> xtype="adql:REGION" datatype="char" arraysize="*"> >>> >>> Service 2 : >> ucd="phys.area;obs.field" unit="deg"> >>> >>> Service 3 : >> arraysize="*"> >>> >>> 3 different services, 3 different ways to express the same thing. >>> Having a unique unambiguous way to detect the STC-S field would be >>> very helpful for client consuming these services. >>> Cheers, >>> >>> Thomas >>> >>> Thomas Boch wrote: >>>> Fran?ois, >>>> >>>> I wish this is something that could be standardized in a forecoming >>>> revision of the SIAP document, so that clients can easily find out >>>> which FIELD support the STC-S description. >>>> >>>> Thomas >>>> >>>> Fran?ois Bonnarel wrote: >>>>> Hi Thomas, >>>>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area >>>>> for this . We have something similar in the SIA2 prototypes >>>>> I guess eventually for sia we will have something like >>>>> sia:Char.SpatialAxis.Coverage.Support.Area >>>>> although the problem of the name space is still to be discussed... >>>>> By the way you can write this field as an STC-S feature ... >>>>> Cheers >>>>> Fran?ois >>>>> Le 29/08/2011 12:02, Thomas Boch a ?crit : >>>>>> Hi, >>>>>> >>>>>> Embedding a STC region in the SIAP query response is becoming more >>>>>> and more widespread, and we would like to support this feature in >>>>>> Aladin. >>>>>> I would like to know if there is a standard (or de facto standard) >>>>>> utype to characterize the FIELD holding the STC region description. >>>>>> Cheers, >>>>>> >>>>>> Thomas >>>>>> >>>>> >>>> >>>> >>> >>> >> Thomas Boch a ?crit : >>> To illustrate my request, here is is how the STC-S FIELD is described >>> in the query response of three different SIAP services : >>> >>> Service 1 : >> xtype="adql:REGION" datatype="char" arraysize="*"> >>> >>> Service 2 : >> ucd="phys.area;obs.field" unit="deg"> >>> >>> Service 3 : >> arraysize="*"> >>> >>> 3 different services, 3 different ways to express the same thing. >>> Having a unique unambiguous way to detect the STC-S field would be >>> very helpful for client consuming these services. >>> Cheers, >>> >>> Thomas >>> >>> Thomas Boch wrote: >>>> Fran?ois, >>>> >>>> I wish this is something that could be standardized in a forecoming >>>> revision of the SIAP document, so that clients can easily find out >>>> which FIELD support the STC-S description. >>>> >>>> Thomas >>>> >>>> Fran?ois Bonnarel wrote: >>>>> Hi Thomas, >>>>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area >>>>> for this . We have something similar in the SIA2 prototypes >>>>> I guess eventually for sia we will have something like >>>>> sia:Char.SpatialAxis.Coverage.Support.Area >>>>> although the problem of the name space is still to be discussed... >>>>> By the way you can write this field as an STC-S feature ... >>>>> Cheers >>>>> Fran?ois >>>>> Le 29/08/2011 12:02, Thomas Boch a ?crit : >>>>>> Hi, >>>>>> >>>>>> Embedding a STC region in the SIAP query response is becoming more >>>>>> and more widespread, and we would like to support this feature in >>>>>> Aladin. >>>>>> I would like to know if there is a standard (or de facto standard) >>>>>> utype to characterize the FIELD holding the STC region description. >>>>>> Cheers, >>>>>> >>>>>> Thomas >>>>>> >>>>> >>>> >>>> >>> >>> >> >> >> -- >> ===================================================================== >> Fran?ois Bonnarel Observatoire Astronomique de Strasbourg >> CDS (Centre de donn?es 11, rue de l'Universit? >> astronomiques de Strasbourg) F--67000 Strasbourg (France) >> >> Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html >> Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr >> --------------------------------------------------------------------- >> > -------------------------------------------------------------------------- > Arnold H. Rots Chandra X-ray Science Center > Smithsonian Astrophysical Observatory tel: +1 617 496 7701 > 60 Garden Street, MS 67 fax: +1 617 495 7356 > Cambridge, MA 02138 arots at head.cfa.harvard.edu > USA http://hea-www.harvard.edu/~arots/ > -------------------------------------------------------------------------- > From francois.bonnarel at astro.unistra.fr Fri Sep 2 03:21:24 2011 From: francois.bonnarel at astro.unistra.fr (Francois Bonnarel) Date: Fri, 02 Sep 2011 12:21:24 +0200 Subject: [Fwd: Re: utype for STC region in SIAP query response] Message-ID: <4E60AE24.3060204@astro.unistra.fr> -- ===================================================================== Fran?ois Bonnarel Observatoire Astronomique de Strasbourg CDS (Centre de donn?es 11, rue de l'Universit? astronomiques de Strasbourg) F--67000 Strasbourg (France) Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr --------------------------------------------------------------------- -------------- next part -------------- An embedded message was scrubbed... From: Francois Bonnarel Subject: Re: utype for STC region in SIAP query response Date: Fri, 02 Sep 2011 12:09:23 +0200 Size: 10771 URL: From arots at head.cfa.harvard.edu Fri Sep 2 06:27:21 2011 From: arots at head.cfa.harvard.edu (Arnold Rots) Date: Fri, 2 Sep 2011 09:27:21 -0400 (EDT) Subject: utype for STC region in SIAP query response In-Reply-To: Message-ID: <201109021327.p82DRM1Y004144@xebec.cfa.harvard.edu> I disagree with this assessment, but won't comment further as there is no eed to rehash past discussions. - Arnold Douglas Tody wrote: > This would be a fine solution, considering only STC; however it is > perfectly legal for a DM to be based upon other models (in a way defined > by the new model such as Char or SIA) and define new, more > application-specific Utypes. There is an especially strong argument for > this for essential elements of the new model so that we have a well > defined and self consistent model. "Falling through" to reference > arbitrary elements of another model, with its own externally defined > namespace, is fine for extension but could significantly expand the > complexity of the new model as a client application might then need to > fully understand multiple models to be able to deal with the data. In > the case here, Char defines strong and consistent concepts for the > measurement axes, Coverage, Support, etc. which are important to the > Char model, and merely uses STC-S to represent the datum in question. > > Regarding UCDs for representing footprints in SIA-1, this can only be > done by extension and there is no standard way to do it at present. We > really need SIAV2 to address this sort of concern. > > - Doug > > > On Thu, 1 Sep 2011, Arnold Rots wrote: > > > Here is what, in my view, is the correct utype for a footprint region: > > > > stc:ObservationLocation.AstroCoordArea.Region > > > > Pretty simple; though one might quibble whether it should be: > > > > stc:ObsDataLocation.ObservationLocation.AstroCoordArea.Region > > > > > > - Arnold > > > > > > Francois Bonnarel wrote: > > [ Charset ISO-8859-1 unsupported, converting... ] > >> Hi Thomas, > >> To complete my answer to Markus > >> we could have something like > >> > >> >> arraysize="*" utype="sia:Char.SpatialAxis.Coverage.Support.Area" > > >> > >> I think something like this can be proposed (if not yet the case) > >> in SIA2... > >> this will be consistent with ObsTap... and could also be included > >> in the next version of SSA et al. > >> > >> For current SIA services, maybe we could have a short > >> recommendation or note to propose using > >> the same as an additional field in the SIA1 Query response > >> Cheers > >> Fran?ois > >>> To illustrate my request, here is is how the STC-S FIELD is described > >>> in the query response of three different SIAP services : > >>> > >>> Service 1 : >>> xtype="adql:REGION" datatype="char" arraysize="*"> > >>> > >>> Service 2 : >>> ucd="phys.area;obs.field" unit="deg"> > >>> > >>> Service 3 : >>> arraysize="*"> > >>> > >>> 3 different services, 3 different ways to express the same thing. > >>> Having a unique unambiguous way to detect the STC-S field would be > >>> very helpful for client consuming these services. > >>> Cheers, > >>> > >>> Thomas > >>> > >>> Thomas Boch wrote: > >>>> Fran?ois, > >>>> > >>>> I wish this is something that could be standardized in a forecoming > >>>> revision of the SIAP document, so that clients can easily find out > >>>> which FIELD support the STC-S description. > >>>> > >>>> Thomas > >>>> > >>>> Fran?ois Bonnarel wrote: > >>>>> Hi Thomas, > >>>>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area > >>>>> for this . We have something similar in the SIA2 prototypes > >>>>> I guess eventually for sia we will have something like > >>>>> sia:Char.SpatialAxis.Coverage.Support.Area > >>>>> although the problem of the name space is still to be discussed... > >>>>> By the way you can write this field as an STC-S feature ... > >>>>> Cheers > >>>>> Fran?ois > >>>>> Le 29/08/2011 12:02, Thomas Boch a ?crit : > >>>>>> Hi, > >>>>>> > >>>>>> Embedding a STC region in the SIAP query response is becoming more > >>>>>> and more widespread, and we would like to support this feature in > >>>>>> Aladin. > >>>>>> I would like to know if there is a standard (or de facto standard) > >>>>>> utype to characterize the FIELD holding the STC region description. > >>>>>> Cheers, > >>>>>> > >>>>>> Thomas > >>>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> To illustrate my request, here is is how the STC-S FIELD is described > >>> in the query response of three different SIAP services : > >>> > >>> Service 1 : >>> xtype="adql:REGION" datatype="char" arraysize="*"> > >>> > >>> Service 2 : >>> ucd="phys.area;obs.field" unit="deg"> > >>> > >>> Service 3 : >>> arraysize="*"> > >>> > >>> 3 different services, 3 different ways to express the same thing. > >>> Having a unique unambiguous way to detect the STC-S field would be > >>> very helpful for client consuming these services. > >>> Cheers, > >>> > >>> Thomas > >>> > >>> Thomas Boch wrote: > >>>> Fran?ois, > >>>> > >>>> I wish this is something that could be standardized in a forecoming > >>>> revision of the SIAP document, so that clients can easily find out > >>>> which FIELD support the STC-S description. > >>>> > >>>> Thomas > >>>> > >>>> Fran?ois Bonnarel wrote: > >>>>> Hi Thomas, > >>>>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area > >>>>> for this . We have something similar in the SIA2 prototypes > >>>>> I guess eventually for sia we will have something like > >>>>> sia:Char.SpatialAxis.Coverage.Support.Area > >>>>> although the problem of the name space is still to be discussed... > >>>>> By the way you can write this field as an STC-S feature ... > >>>>> Cheers > >>>>> Fran?ois > >>>>> Le 29/08/2011 12:02, Thomas Boch a ?crit : > >>>>>> Hi, > >>>>>> > >>>>>> Embedding a STC region in the SIAP query response is becoming more > >>>>>> and more widespread, and we would like to support this feature in > >>>>>> Aladin. > >>>>>> I would like to know if there is a standard (or de facto standard) > >>>>>> utype to characterize the FIELD holding the STC region description. > >>>>>> Cheers, > >>>>>> > >>>>>> Thomas > >>>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> Thomas Boch a ?crit : > >>> To illustrate my request, here is is how the STC-S FIELD is described > >>> in the query response of three different SIAP services : > >>> > >>> Service 1 : >>> xtype="adql:REGION" datatype="char" arraysize="*"> > >>> > >>> Service 2 : >>> ucd="phys.area;obs.field" unit="deg"> > >>> > >>> Service 3 : >>> arraysize="*"> > >>> > >>> 3 different services, 3 different ways to express the same thing. > >>> Having a unique unambiguous way to detect the STC-S field would be > >>> very helpful for client consuming these services. > >>> Cheers, > >>> > >>> Thomas > >>> > >>> Thomas Boch wrote: > >>>> Fran?ois, > >>>> > >>>> I wish this is something that could be standardized in a forecoming > >>>> revision of the SIAP document, so that clients can easily find out > >>>> which FIELD support the STC-S description. > >>>> > >>>> Thomas > >>>> > >>>> Fran?ois Bonnarel wrote: > >>>>> Hi Thomas, > >>>>> Obstap has obs:Char.SpatialAxis.Coverage.Support.Area > >>>>> for this . We have something similar in the SIA2 prototypes > >>>>> I guess eventually for sia we will have something like > >>>>> sia:Char.SpatialAxis.Coverage.Support.Area > >>>>> although the problem of the name space is still to be discussed... > >>>>> By the way you can write this field as an STC-S feature ... > >>>>> Cheers > >>>>> Fran?ois > >>>>> Le 29/08/2011 12:02, Thomas Boch a ?crit : > >>>>>> Hi, > >>>>>> > >>>>>> Embedding a STC region in the SIAP query response is becoming more > >>>>>> and more widespread, and we would like to support this feature in > >>>>>> Aladin. > >>>>>> I would like to know if there is a standard (or de facto standard) > >>>>>> utype to characterize the FIELD holding the STC region description. > >>>>>> Cheers, > >>>>>> > >>>>>> Thomas > >>>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> -- > >> ===================================================================== > >> Fran?ois Bonnarel Observatoire Astronomique de Strasbourg > >> CDS (Centre de donn?es 11, rue de l'Universit? > >> astronomiques de Strasbourg) F--67000 Strasbourg (France) > >> > >> Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html > >> Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr > >> --------------------------------------------------------------------- > >> > > -------------------------------------------------------------------------- > > Arnold H. Rots Chandra X-ray Science Center > > Smithsonian Astrophysical Observatory tel: +1 617 496 7701 > > 60 Garden Street, MS 67 fax: +1 617 495 7356 > > Cambridge, MA 02138 arots at head.cfa.harvard.edu > > USA http://hea-www.harvard.edu/~arots/ > > -------------------------------------------------------------------------- > > > -------------------------------------------------------------------------- Arnold H. Rots Chandra X-ray Science Center Smithsonian Astrophysical Observatory tel: +1 617 496 7701 60 Garden Street, MS 67 fax: +1 617 495 7356 Cambridge, MA 02138 arots at head.cfa.harvard.edu USA http://hea-www.harvard.edu/~arots/ -------------------------------------------------------------------------- From msdemlei at ari.uni-heidelberg.de Mon Sep 5 06:17:46 2011 From: msdemlei at ari.uni-heidelberg.de (Markus Demleitner) Date: Mon, 5 Sep 2011 15:17:46 +0200 Subject: utype for STC region in SIAP query response In-Reply-To: <4E5F4D73.50904@astro.unistra.fr> References: <4E5B63BC.6030009@astro.unistra.fr> <4E5B8657.7010607@astro.unistra.fr> <4E5B8996.2070709@astro.unistra.fr> <4E5CB292.1090307@astro.unistra.fr> <4E5F4D73.50904@astro.unistra.fr> Message-ID: <20110905131746.GA23945@ari.uni-heidelberg.de> Dear Colleagues, On Thu, Sep 01, 2011 at 11:16:35AM +0200, Francois Bonnarel wrote: > Hi Thomas, > To complete my answer to Markus > we could have something like > arraysize="*" utype="sia:Char.SpatialAxis.Coverage.Support.Area" > > I think something like this can be proposed (if not yet the > case) in SIA2... > this will be consistent with ObsTap... and could also be > included in the next version of SSA et al. Allow me to once again advertise the use of "standard" STC groups[1] -- while it's fine (and indeed necessary in many cases) for protocol-specific data models have their own utypes for STC data, multiprotocol clients greatly benefit if fields containing STC information are marked up as such using the STC data model itself. And the current problem -- identifying which field in a SIAP response contains a region -- would be more or less solved just based on existing IVOA standards. So, I'd suggest to recommend including an STC group to SIAP responses (which, actually, is already implicitely done by the VOTable spec). For SIAP, this is pretty much boilerplate and looks like this for my SIAP services (fix the refs as necessary for you, and you may need to adapt the coverage's utype): To identify the column containing the region, you'd look for (probably in this sequence, xpath syntax): GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Polygon"] GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Convex"] GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Ellipse"] GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Circle"] GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Box"] -- not too simple, but still preferable to having to know every data model. Moral: The rotten substitution groups in STC-X again mess things up royally, and it would have been much nicer if STC-X had just defined, say, Region and an attribute shape (or something), but it's too late to cry over spilled milk now. Since I can't see anybody moving to review STC and its serializations, I'd plea to use what we have rather than to invent new workarounds every time we encounter an STC-related problem. Cheers, Markus [1] Ok, while the STC data model is a recommendation, the specific form of the group is just a note, http://www.ivoa.net/Documents/Notes/VOTableSTC/ (as is STC-X) -- but the note hopefully reflects a wide consensus among the interested parties. From dtody at nrao.edu Mon Sep 5 15:16:02 2011 From: dtody at nrao.edu (Douglas Tody) Date: Mon, 5 Sep 2011 16:16:02 -0600 (MDT) Subject: utype for STC region in SIAP query response In-Reply-To: <20110905131746.GA23945@ari.uni-heidelberg.de> References: <4E5B63BC.6030009@astro.unistra.fr> <4E5B8657.7010607@astro.unistra.fr> <4E5B8996.2070709@astro.unistra.fr> <4E5CB292.1090307@astro.unistra.fr> <4E5F4D73.50904@astro.unistra.fr> <20110905131746.GA23945@ari.uni-heidelberg.de> Message-ID: Hi - The extension mechanism used in the DAL services allows pass through of arbitrary metadata so this is technically legal, and could work to pass arbitrary STC metadata in cases where some feature is needed which is not addressed by the standard protocol. However in cases which are directly addressed by the protocol, the standard protocol mechanisms should be used instead. Otherwise we no longer have a standard protocol and a client application will not know what to look for (no existing clients will be able to use this feature of your service Markus). In this case the Char model which is used in all the DAL services (SSA, ObsTAP/ObsCoreDM, SIAV2, and soon SED, TimeSeries) provides a mechanism for describing coverage in the different measurement axes, and this is what should be used for these protocols. - Doug On Mon, 5 Sep 2011, Markus Demleitner wrote: > Dear Colleagues, > > On Thu, Sep 01, 2011 at 11:16:35AM +0200, Francois Bonnarel wrote: >> Hi Thomas, >> To complete my answer to Markus >> we could have something like >> > arraysize="*" utype="sia:Char.SpatialAxis.Coverage.Support.Area" > >> I think something like this can be proposed (if not yet the >> case) in SIA2... >> this will be consistent with ObsTap... and could also be >> included in the next version of SSA et al. > > Allow me to once again advertise the use of "standard" STC groups[1] -- > while it's fine (and indeed necessary in many cases) for > protocol-specific data models have their own utypes for STC data, > multiprotocol clients greatly benefit if fields containing STC > information are marked up as such using the STC data model itself. > And the current problem -- identifying which field in a SIAP response > contains a region -- would be more or less solved just based on > existing IVOA standards. > > So, I'd suggest to recommend including an STC group to SIAP responses > (which, actually, is already implicitely done by the VOTable spec). > For SIAP, this is pretty much boilerplate and looks like this for my > SIAP services (fix the refs as necessary for you, and you may need to > adapt the coverage's utype): > > > utype="stc:AstroCoordSystem.SpaceFrame.CoordFlavor" > arraysize="*" value="SPHERICAL" name="CoordFlavor"/> > utype="stc:AstroCoordSystem.SpaceFrame.CoordRefFrame" > arraysize="*" value="ICRS" name="CoordRefFrame"/> > utype="stc:AstroCoordSystem.TimeFrame.TimeScale" > arraysize="*" value="TT" name="TimeScale"/> > value="http://www.ivoa.net/xml/STC/stc-v1.30.xsd" name="URI"/> > > ref="bandpassHi"/> > ref="bandpassLo"/> > ref="centerAlpha"/> > ref="centerDelta"/> > > > > > To identify the column containing the region, you'd look for > (probably in this sequence, xpath syntax): > > GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Polygon"] > GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Convex"] > GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Ellipse"] > GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Circle"] > GROUP[@utype='stc:CatalogEntryLocation']/FIELDref[@utype="stc:AstroCoordArea.Box"] > > -- not too simple, but still preferable to having to know every data > model. > > Moral: The rotten substitution groups in STC-X again mess things up > royally, and it would have been much nicer if STC-X had just defined, > say, Region and an attribute shape (or something), but it's too late > to cry over spilled milk now. Since I can't see anybody moving to > review STC and its serializations, I'd plea to use what we have > rather than to invent new workarounds every time we encounter an > STC-related problem. > > Cheers, > > Markus > > > [1] Ok, while the STC data model is a recommendation, the specific > form of the group is just a note, > http://www.ivoa.net/Documents/Notes/VOTableSTC/ (as is STC-X) -- but > the note hopefully reflects a wide consensus among the interested > parties. > From msdemlei at ari.uni-heidelberg.de Thu Sep 8 00:02:54 2011 From: msdemlei at ari.uni-heidelberg.de (Markus Demleitner) Date: Thu, 8 Sep 2011 09:02:54 +0200 Subject: utype for STC region in SIAP query response In-Reply-To: References: <4E5B63BC.6030009@astro.unistra.fr> <4E5B8657.7010607@astro.unistra.fr> <4E5B8996.2070709@astro.unistra.fr> <4E5CB292.1090307@astro.unistra.fr> <4E5F4D73.50904@astro.unistra.fr> <20110905131746.GA23945@ari.uni-heidelberg.de> Message-ID: <20110908070254.GA27732@ari.uni-heidelberg.de> Hi Doug, Hi list, On Mon, Sep 05, 2011 at 04:16:02PM -0600, Douglas Tody wrote: > directly addressed by the protocol, the standard protocol mechanisms > should be used instead. Otherwise we no longer have a standard protocol > and a client application will not know what to look for (no existing > clients will be able to use this feature of your service Markus). In The STC embedding has deliberately been designed to not clobber the FIELD's utype attribute, so there's no technical reason for "instead" (and, apart from a moderate amount of work, no other I can see either). However, I still maintain there's a good reason to include standard STC metadata whether or not a specific data model has mechanisms of its own: Generic VOTable consumers do not need to know lots of data models and utypes to make sense of coordinates. That each protocol/DM will have different utype prefixes doesn't help either. That no client so far can make sense of, erm, "standard" STC metadata is regrettable, but that's only going to change when the data providers start embedding such metadata. The situation right now is that outside of some protocols, we only have a deprecated (and crude) mechanism to communicate STC metadata in VOTables (COOSYS) -- don't you agree that we should change this? Cheers, Markus From patrick.dowler at nrc-cnrc.gc.ca Thu Sep 15 14:38:39 2011 From: patrick.dowler at nrc-cnrc.gc.ca (Patrick Dowler) Date: Thu, 15 Sep 2011 14:38:39 -0700 Subject: DAL sessions - Pune 2011 Message-ID: <201109151438.39531.patrick.dowler@nrc-cnrc.gc.ca> The program for the interop includes 4 sessions for DAL, each with just one topic in order to have plenty of time for discussion. TAPRegExt DALI DataLink PQL In addition, there is a Theory/DAL session on SimDAL which should not conflict with the above and I'm pretty sure TAPRegExt will not conflict with any Registry-WG sessions. I am not sure at this point exactly what format would be best, but since there has been no visible progress on DALI, DataLink, or PQL (we're all very busy, it seems :-) I think the best thing would be to simply have an open discussion, try to dig into the details, and explore all the options. If anyone has done any prototyping of these things, we could all benefit from a short summary of the experiences. I will be creating a wiki page for each topic (for those that don't exist), linking to them from the main DAL wiki page, and linking to them from the interop program. At that point, interested people can just add discussion points to a list and we'll use that as the session agenda. For now, let me know if you would like to report on prototyping experiences related to the above topics and/or if you have any other ideas on how best to use the time we have to get these things rolling. Pat Dowler (DAL chair) Mike Fitzpatrick (DAL vice-chair) -- Patrick Dowler Tel/T?l: (250) 363-0044 Canadian Astronomy Data Centre National Research Council Canada 5071 West Saanich Road Victoria, BC V9E 2M7 Centre canadien de donnees astronomiques Conseil national de recherches Canada 5071, chemin West Saanich Victoria (C.-B.) V9E 2M7 From rthomp at stsci.edu Thu Sep 22 09:52:28 2011 From: rthomp at stsci.edu (Randy Thompson) Date: Thu, 22 Sep 2011 16:52:28 +0000 Subject: Support for data containing NaN values Message-ID: As a general question, does data containing NaN values violate any VO standards or protocols,and if not, should VO applications be expected to accept them as input? Randy From m.b.taylor at bristol.ac.uk Mon Sep 26 02:06:08 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Mon, 26 Sep 2011 10:06:08 +0100 (BST) Subject: Support for data containing NaN values In-Reply-To: References: Message-ID: On Thu, 22 Sep 2011, Randy Thompson wrote: > As a general question, does data containing NaN values > violate any VO standards or protocols,and if not, should VO > applications be expected to accept them as input? the question is rather broad ("data" can take many forms), but on the whole the answer is that most standards and software in the VO should and do behave sensibly in the presence of NaN-valued floating point values. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From fitz at noao.edu Mon Sep 26 02:40:56 2011 From: fitz at noao.edu (Mike Fitzpatrick) Date: Mon, 26 Sep 2011 02:40:56 -0700 Subject: Support for data containing NaN values In-Reply-To: References: Message-ID: I took the question differently: If VO allows FITS data, and FITS data allows NaN, then apps should of course allow for this. OTOH, if the question is whether "VO data" as is serialized in a VOTable allows NaN values the I think the answer is 'no' (but I'd have to check). There are similar issues with how NULL values are handled, but again it depends on whether it is in the serialized VOTable or the end data product being accessed. The DAL protocols don't say anything about NaN/NULL beyond how they might be serialized, FITS is FITS and if that's what the app retrieves in the end and then that is the standard to follow when interpreting the data. Was that your question? My $0.02, -Mike On Mon, Sep 26, 2011 at 2:06 AM, Mark Taylor wrote: > On Thu, 22 Sep 2011, Randy Thompson wrote: > > > As a general question, does data containing NaN values > > violate any VO standards or protocols,and if not, should VO > > applications be expected to accept them as input? > > the question is rather broad ("data" can take many forms), but on > the whole the answer is that most standards and software in the VO > should and do behave sensibly in the presence of NaN-valued floating > point values. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.b.taylor at bristol.ac.uk Mon Sep 26 02:53:18 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Mon, 26 Sep 2011 10:53:18 +0100 (BST) Subject: Support for data containing NaN values In-Reply-To: References: Message-ID: Mike, in fact VOTable does permit NaN values in (single or double) floating point data; I think this is a consequence of the design decision that the VOTable model for data is to be as close as possible to that of FITS. Furthermore, NaN is how VOTable recommends to represent NULL values in floating point data (again, following FITS) - whether that's a good idea or not is a question that has been debated elsewhere, but that's what VOTable section 6 says (http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC41) Mark On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: > I took the question differently: If VO allows FITS data, and > FITS data allows NaN, then apps should of course allow > for this. OTOH, if the question is whether "VO data" as is > serialized in a VOTable allows NaN values the I think the > answer is 'no' (but I'd have to check). There are similar issues > with how NULL values are handled, but again it depends on > whether it is in the serialized VOTable or the end data product > being accessed. The DAL protocols don't say anything about > NaN/NULL beyond how they might be serialized, FITS is FITS > and if that's what the app retrieves in the end and then that is > the standard to follow when interpreting the data. Was that > your question? > > My $0.02, > -Mike > > > > On Mon, Sep 26, 2011 at 2:06 AM, Mark Taylor wrote: > > > On Thu, 22 Sep 2011, Randy Thompson wrote: > > > > > As a general question, does data containing NaN values > > > violate any VO standards or protocols,and if not, should VO > > > applications be expected to accept them as input? > > > > the question is rather broad ("data" can take many forms), but on > > the whole the answer is that most standards and software in the VO > > should and do behave sensibly in the presence of NaN-valued floating > > point values. > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Thomas.A.McGlynn at nasa.gov Mon Sep 26 05:33:43 2011 From: Thomas.A.McGlynn at nasa.gov (Tom McGlynn) Date: Mon, 26 Sep 2011 08:33:43 -0400 Subject: Support for data containing NaN values In-Reply-To: References: Message-ID: <4E807127.3020908@nasa.gov> I'm not sure that Mark's comments really addresses the Randy's question. Suppose we have an original dataset, O, in some non-VO format and a VO serialization of this dataset, V. Both O and V may contain NaNs. As Mark points out a NaN in V is the recommended representation for a null value. So in any context in which null values are distinct from NaNs, VOTables cannot distinguish them, i.e., a NaN in the VOTable does not in general mean that there was a NaN in the original data. So if you wish to preserve NaNs VOTables are not currently a safe way to do so. As alluded to, this particular issue has been discussed before and some thoughts have been collected at http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues. Tom Mark Taylor wrote: > Mike, > > in fact VOTable does permit NaN values in (single or double) floating > point data; I think this is a consequence of the design decision > that the VOTable model for data is to be as close as possible to > that of FITS. Furthermore, NaN is how VOTable recommends to represent > NULL values in floating point data (again, following FITS) - whether > that's a good idea or not is a question that has been debated elsewhere, > but that's what VOTable section 6 says > (http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC41) > > Mark > > On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: > >> I took the question differently: If VO allows FITS data, and >> FITS data allows NaN, then apps should of course allow >> for this. OTOH, if the question is whether "VO data" as is >> serialized in a VOTable allows NaN values the I think the >> answer is 'no' (but I'd have to check). There are similar issues >> with how NULL values are handled, but again it depends on >> whether it is in the serialized VOTable or the end data product >> being accessed. The DAL protocols don't say anything about >> NaN/NULL beyond how they might be serialized, FITS is FITS >> and if that's what the app retrieves in the end and then that is >> the standard to follow when interpreting the data. Was that >> your question? >> >> My $0.02, >> -Mike >> >> >> >> On Mon, Sep 26, 2011 at 2:06 AM, Mark Taylorwrote: >> >>> On Thu, 22 Sep 2011, Randy Thompson wrote: >>> >>>> As a general question, does data containing NaN values >>>> violate any VO standards or protocols,and if not, should VO >>>> applications be expected to accept them as input? >>> >>> the question is rather broad ("data" can take many forms), but on >>> the whole the answer is that most standards and software in the VO >>> should and do behave sensibly in the presence of NaN-valued floating >>> point values. >>> >>> -- >>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From rthomp at stsci.edu Mon Sep 26 06:24:47 2011 From: rthomp at stsci.edu (Randy Thompson) Date: Mon, 26 Sep 2011 13:24:47 +0000 Subject: Support for data containing NaN values In-Reply-To: <4E807127.3020908@nasa.gov> Message-ID: Thanks for the replies. The question initially came up in testing a VAO tool which will convert various input file formats to "compliant" VOTable or FITS files. If NaNs are included in the input files the values are passed unchanged. It sounds like this does not violate any VO standards, and other VO applications should be expected to handle them in data contained in either FITS or VOTable format. We are also archiving FITS files from a project using -inf values to flag bad data points and wondered if any further processing would be required before these files would be considered "VO compliant". Randy On 9/26/11 8:33 AM, "Tom McGlynn" wrote: >I'm not sure that Mark's comments really addresses the Randy's >question. Suppose we have an original dataset, O, in some non-VO >format and a VO serialization of this dataset, V. Both O and V may >contain NaNs. As Mark points out a NaN in V is the recommended >representation for a null value. So in any context in which null >values are distinct from NaNs, VOTables cannot distinguish them, i.e., >a NaN in the VOTable does not in general mean that there was a NaN in >the original data. So if you wish to preserve NaNs VOTables are not >currently a safe way to do so. > >As alluded to, this particular issue has been discussed before and >some thoughts have been collected at >http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues. > > Tom > >Mark Taylor wrote: >> Mike, >> >> in fact VOTable does permit NaN values in (single or double) floating >> point data; I think this is a consequence of the design decision >> that the VOTable model for data is to be as close as possible to >> that of FITS. Furthermore, NaN is how VOTable recommends to represent >> NULL values in floating point data (again, following FITS) - whether >> that's a good idea or not is a question that has been debated elsewhere, >> but that's what VOTable section 6 says >> >>(http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC4 >>1) >> >> Mark >> >> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: >> >>> I took the question differently: If VO allows FITS data, and >>> FITS data allows NaN, then apps should of course allow >>> for this. OTOH, if the question is whether "VO data" as is >>> serialized in a VOTable allows NaN values the I think the >>> answer is 'no' (but I'd have to check). There are similar issues >>> with how NULL values are handled, but again it depends on >>> whether it is in the serialized VOTable or the end data product >>> being accessed. The DAL protocols don't say anything about >>> NaN/NULL beyond how they might be serialized, FITS is FITS >>> and if that's what the app retrieves in the end and then that is >>> the standard to follow when interpreting the data. Was that >>> your question? >>> >>> My $0.02, >>> -Mike >>> >>> >>> >>> On Mon, Sep 26, 2011 at 2:06 AM, Mark >>>Taylorwrote: >>> >>>> On Thu, 22 Sep 2011, Randy Thompson wrote: >>>> >>>>> As a general question, does data containing NaN values >>>>> violate any VO standards or protocols,and if not, should VO >>>>> applications be expected to accept them as input? >>>> >>>> the question is rather broad ("data" can take many forms), but on >>>> the whole the answer is that most standards and software in the VO >>>> should and do behave sensibly in the presence of NaN-valued floating >>>> point values. >>>> >>>> -- >>>> Mark Taylor Astronomical Programmer Physics, Bristol University, >>>>UK >>>> m.b.taylor at bris.ac.uk +44-117-928-8776 >>>>http://www.star.bris.ac.uk/~mbt/ >>>> >>> >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor at bristol.ac.uk Mon Sep 26 06:40:40 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Mon, 26 Sep 2011 14:40:40 +0100 (BST) Subject: Support for data containing NaN values In-Reply-To: References: Message-ID: Regarding -inf, I don't think that there is any special VO compliance issue here, but I would question the idea of using -inf to flag bad values, unless perhaps the data is such that absence of data can be presumed to imply a large negative value. While -inf is better than, say, 9999, it is likely to cause problems when processing data to do something like assess its range or calculate statistics. In the absence of a way of flagging a true null value, NaN would seem like a less problematic choice. Just my $0.02. Mark On Mon, 26 Sep 2011, Randy Thompson wrote: > Thanks for the replies. The question initially came up in testing a > VAO tool which will convert various input file formats to "compliant" > VOTable or FITS files. If NaNs are included in the input files the > values are passed unchanged. It sounds like this does not violate > any VO standards, and other VO applications should be expected to > handle them in data contained in either FITS or VOTable format. > > We are also archiving FITS files from a project using -inf values > to flag bad data points and wondered if any further processing would > be required before these files would be considered "VO compliant". > > Randy > > > On 9/26/11 8:33 AM, "Tom McGlynn" wrote: > > >I'm not sure that Mark's comments really addresses the Randy's > >question. Suppose we have an original dataset, O, in some non-VO > >format and a VO serialization of this dataset, V. Both O and V may > >contain NaNs. As Mark points out a NaN in V is the recommended > >representation for a null value. So in any context in which null > >values are distinct from NaNs, VOTables cannot distinguish them, i.e., > >a NaN in the VOTable does not in general mean that there was a NaN in > >the original data. So if you wish to preserve NaNs VOTables are not > >currently a safe way to do so. > > > >As alluded to, this particular issue has been discussed before and > >some thoughts have been collected at > >http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues. > > > > Tom > > > >Mark Taylor wrote: > >> Mike, > >> > >> in fact VOTable does permit NaN values in (single or double) floating > >> point data; I think this is a consequence of the design decision > >> that the VOTable model for data is to be as close as possible to > >> that of FITS. Furthermore, NaN is how VOTable recommends to represent > >> NULL values in floating point data (again, following FITS) - whether > >> that's a good idea or not is a question that has been debated elsewhere, > >> but that's what VOTable section 6 says > >> > >>(http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC4 > >>1) > >> > >> Mark > >> > >> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: > >> > >>> I took the question differently: If VO allows FITS data, and > >>> FITS data allows NaN, then apps should of course allow > >>> for this. OTOH, if the question is whether "VO data" as is > >>> serialized in a VOTable allows NaN values the I think the > >>> answer is 'no' (but I'd have to check). There are similar issues > >>> with how NULL values are handled, but again it depends on > >>> whether it is in the serialized VOTable or the end data product > >>> being accessed. The DAL protocols don't say anything about > >>> NaN/NULL beyond how they might be serialized, FITS is FITS > >>> and if that's what the app retrieves in the end and then that is > >>> the standard to follow when interpreting the data. Was that > >>> your question? > >>> > >>> My $0.02, > >>> -Mike > >>> > >>> > >>> > >>> On Mon, Sep 26, 2011 at 2:06 AM, Mark > >>>Taylorwrote: > >>> > >>>> On Thu, 22 Sep 2011, Randy Thompson wrote: > >>>> > >>>>> As a general question, does data containing NaN values > >>>>> violate any VO standards or protocols,and if not, should VO > >>>>> applications be expected to accept them as input? > >>>> > >>>> the question is rather broad ("data" can take many forms), but on > >>>> the whole the answer is that most standards and software in the VO > >>>> should and do behave sensibly in the presence of NaN-valued floating > >>>> point values. > >>>> > >>>> -- > >>>> Mark Taylor Astronomical Programmer Physics, Bristol University, > >>>>UK > >>>> m.b.taylor at bris.ac.uk +44-117-928-8776 > >>>>http://www.star.bris.ac.uk/~mbt/ > >>>> > >>> > >> > >> -- > >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK > >> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From francois at cdsarc.u-strasbg.fr Mon Sep 26 07:44:44 2011 From: francois at cdsarc.u-strasbg.fr (Francois Ochsenbein (ext.52429)) Date: Mon, 26 Sep 2011 16:44:44 +0200 Subject: Support for data containing NaN values In-Reply-To: <4E807127.3020908@nasa.gov> References: <4E807127.3020908@nasa.gov> Message-ID: <20110926144444.113EA25EA3@cdsarc.u-strasbg.fr> Hi Tom, Just 2 remarks: * have you in mind any practical case in which a distinction between NaN (not-a-number IEEE pattern) and NULL (in relational database jargon) is meaningful ? * do you know any relational database which accepts to enter NaN (and/or +/-Inf) values ? As far as I know, these IEEE floating-point patterns are not handled (the relational databases I use generate arithmetic errors, and refuse the operation which could lead to NaN's or Inf's) And yes, the VOTable document assumes (as FITS does) that NaN and NULL have the same semantic meaning -- i.e. that the answer is "no" to both questions. Cheers, francois > >I'm not sure that Mark's comments really addresses the Randy's >question. Suppose we have an original dataset, O, in some non-VO >format and a VO serialization of this dataset, V. Both O and V may >contain NaNs. As Mark points out a NaN in V is the recommended >representation for a null value. So in any context in which null >values are distinct from NaNs, VOTables cannot distinguish them, i.e., >a NaN in the VOTable does not in general mean that there was a NaN in >the original data. So if you wish to preserve NaNs VOTables are not >currently a safe way to do so. > >As alluded to, this particular issue has been discussed before and >some thoughts have been collected at >http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues. > > Tom > >Mark Taylor wrote: >> Mike, >> >> in fact VOTable does permit NaN values in (single or double) floating >> point data; I think this is a consequence of the design decision >> that the VOTable model for data is to be as close as possible to >> that of FITS. Furthermore, NaN is how VOTable recommends to represent >> NULL values in floating point data (again, following FITS) - whether >> that's a good idea or not is a question that has been debated elsewhere, >> but that's what VOTable section 6 says >> (http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC41) >> >> Mark >> >> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: >> >>> I took the question differently: If VO allows FITS data, and >>> FITS data allows NaN, then apps should of course allow >>> for this. OTOH, if the question is whether "VO data" as is >>> serialized in a VOTable allows NaN values the I think the >>> answer is 'no' (but I'd have to check). There are similar issues >>> with how NULL values are handled, but again it depends on >>> whether it is in the serialized VOTable or the end data product >>> being accessed. The DAL protocols don't say anything about >>> NaN/NULL beyond how they might be serialized, FITS is FITS >>> and if that's what the app retrieves in the end and then that is >>> the standard to follow when interpreting the data. Was that >>> your question? >>> >>> My $0.02, >>> -Mike >>> >>> >>> >>> On Mon, Sep 26, 2011 at 2:06 AM, Mark Taylorwrote: >>> >>>> On Thu, 22 Sep 2011, Randy Thompson wrote: >>>> >>>>> As a general question, does data containing NaN values >>>>> violate any VO standards or protocols,and if not, should VO >>>>> applications be expected to accept them as input? >>>> >>>> the question is rather broad ("data" can take many forms), but on >>>> the whole the answer is that most standards and software in the VO >>>> should and do behave sensibly in the presence of NaN-valued floating >>>> point values. >>>> >>>> -- >>>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>>> >>> >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ ======================================================================= Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 Email: francois at astro.u-strasbg.fr (France) Fax: +33-(0)368 85 24 17 ======================================================================= From Thomas.A.McGlynn at nasa.gov Mon Sep 26 08:37:38 2011 From: Thomas.A.McGlynn at nasa.gov (Tom McGlynn) Date: Mon, 26 Sep 2011 11:37:38 -0400 Subject: Support for data containing NaN values In-Reply-To: <20110926144444.113EA25EA3@cdsarc.u-strasbg.fr> References: <4E807127.3020908@nasa.gov> <20110926144444.113EA25EA3@cdsarc.u-strasbg.fr> Message-ID: <4E809C42.2000902@nasa.gov> Hi Francois, Further comments in answer to your questions. Tom Francois Ochsenbein (ext.52429) wrote: > > Hi Tom, > > Just 2 remarks: > > * have you in mind any practical case in which a distinction > between NaN (not-a-number IEEE pattern) and NULL (in relational > database jargon) is meaningful ? I tend to think of NaN's as expressing a concept that something is 'wrong' while Null's represent that something is 'missing'. E.g., in a telemetry stream I might use a NaN when the instrument is in some invalid state and Null when the instrument is off. Developing this example a little: Suppose I want to have a set of photometric measurements in a standard time frame from an instrument. In the calibrated photometry, Null brightness values indicate the instrument is off, NaN's that the data is invalid. If I feel that I can do the calibration a bit better I might want to look for the NaN rows to see if I can get around the problems that occurred in the original calibration. If the data has been passed through a VOTable somewhere in the processing, the distinction between the null and NaN rows has been lost so I can't easily pick out the 'invalid' values. Sure it would be possible to use a different representation...but this is a reasonable approach and we should be very chary of putting limits on what data users can represent or saying how they have to represent it in what is intended to be a generic format. I think that lack of full support for IEEE floating point numbers is a substantial limit for VOTables. The fact that a standard developed in the 1970's has the same limit isn't really a good reason that we need to propagate the constraint. > > * do you know any relational database which accepts to enter > NaN (and/or +/-Inf) values ? As far as I know, these IEEE > floating-point patterns are not handled (the relational databases > I use generate arithmetic errors, and refuse the operation which > could lead to NaN's or Inf's) > It is my understanding (and personal experience in the first case) that Postgres, Oracle and Ingres support NaNs and Infs. As far as I can tell Sybase, MySQL and SQLServer do not. (See, e.g., 8.1.3 in http://www.postgresql.org/docs/8.2/static/datatype-numeric.html or http://www.techonthenet.com/oracle/functions/nanvl.php) I'm not clear if databases that do not allow NaNs as internal values in databases also preclude them as calculated results, e.g., what the result of select sqrt(-1.) is. > And yes, the VOTable document assumes (as FITS does) that NaN > and NULL have the same semantic meaning -- i.e. that the answer > is "no" to both questions. > > Cheers, francois > >> >> I'm not sure that Mark's comments really addresses the Randy's >> question. Suppose we have an original dataset, O, in some non-VO >> format and a VO serialization of this dataset, V. Both O and V may >> contain NaNs. As Mark points out a NaN in V is the recommended >> representation for a null value. So in any context in which null >> values are distinct from NaNs, VOTables cannot distinguish them, i.e., >> a NaN in the VOTable does not in general mean that there was a NaN in >> the original data. So if you wish to preserve NaNs VOTables are not >> currently a safe way to do so. >> >> As alluded to, this particular issue has been discussed before and >> some thoughts have been collected at >> http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues. >> >> Tom >> >> Mark Taylor wrote: >>> Mike, >>> >>> in fact VOTable does permit NaN values in (single or double) floating >>> point data; I think this is a consequence of the design decision >>> that the VOTable model for data is to be as close as possible to >>> that of FITS. Furthermore, NaN is how VOTable recommends to represent >>> NULL values in floating point data (again, following FITS) - whether >>> that's a good idea or not is a question that has been debated elsewhere, >>> but that's what VOTable section 6 says >>> (http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC41) >>> >>> Mark >>> >>> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: >>> >>>> I took the question differently: If VO allows FITS data, and >>>> FITS data allows NaN, then apps should of course allow >>>> for this. OTOH, if the question is whether "VO data" as is >>>> serialized in a VOTable allows NaN values the I think the >>>> answer is 'no' (but I'd have to check). There are similar issues >>>> with how NULL values are handled, but again it depends on >>>> whether it is in the serialized VOTable or the end data product >>>> being accessed. The DAL protocols don't say anything about >>>> NaN/NULL beyond how they might be serialized, FITS is FITS >>>> and if that's what the app retrieves in the end and then that is >>>> the standard to follow when interpreting the data. Was that >>>> your question? >>>> >>>> My $0.02, >>>> -Mike >>>> >>>> >>>> >>>> On Mon, Sep 26, 2011 at 2:06 AM, Mark Taylorwrote: >>>> >>>>> On Thu, 22 Sep 2011, Randy Thompson wrote: >>>>> >>>>>> As a general question, does data containing NaN values >>>>>> violate any VO standards or protocols,and if not, should VO >>>>>> applications be expected to accept them as input? >>>>> >>>>> the question is rather broad ("data" can take many forms), but on >>>>> the whole the answer is that most standards and software in the VO >>>>> should and do behave sensibly in the presence of NaN-valued floating >>>>> point values. >>>>> >>>>> -- >>>>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>>>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>>>> >>>> >>> >>> -- >>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > ======================================================================= > Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg > 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 > Email: francois at astro.u-strasbg.fr (France) Fax: +33-(0)368 85 24 17 > ======================================================================= From francois at cdsarc.u-strasbg.fr Mon Sep 26 16:24:40 2011 From: francois at cdsarc.u-strasbg.fr (Francois Ochsenbein (ext.52429)) Date: Tue, 27 Sep 2011 01:24:40 +0200 Subject: Support for data containing NaN values In-Reply-To: <4E809C42.2000902@nasa.gov> References: <4E807127.3020908@nasa.gov> <20110926144444.113EA25EA3@cdsarc.u-strasbg.fr> <4E809C42.2000902@nasa.gov> Message-ID: <20110926232441.0948225EA3@cdsarc.u-strasbg.fr> Hi Tom, I think I understand the difference you make between "wrong" and "unspecified"; however the situation is already clumpsy with the lack of well-defined "null" value in the case of integer numbers (yes, the "unspecified" value has to be accurately defined...) . And since the databases are able to understand only a single "null" specification, I would really encourage to avoid further distinctions for the time being: new semantic definitions would need to be defined in FITS (not only in VOTable!) and can't currently be ported to relational databases. By the way I tried in Postgres to apply arithmetic operations between tables with NaN values or +/-Inf; the operations are not always working, e.g. 1/0 or 1/Inf generate an error. For instance with the table ==> select * from t; n | v ----+----------- 0 | -1 -1 | NaN 9 | Infinity -1 | Infinity 0 | -Infinity ==> select n, 1/v from t; ERROR: value out of range: underflow Cheers, francois > >Hi Francois, > >Further comments in answer to your questions. > Tom > >Francois Ochsenbein (ext.52429) wrote: >> >> Hi Tom, >> >> Just 2 remarks: >> >> * have you in mind any practical case in which a distinction >> between NaN (not-a-number IEEE pattern) and NULL (in relational >> database jargon) is meaningful ? > >I tend to think of NaN's as expressing a concept that something is >'wrong' while Null's represent that something is 'missing'. E.g., in >a telemetry stream I might use a NaN when the instrument is in some >invalid state and Null when the instrument is off. Developing this >example a little: Suppose I want to have a set of photometric >measurements in a standard time frame from an instrument. In the >calibrated photometry, Null brightness values indicate the instrument >is off, NaN's that the data is invalid. If I feel that I can do the >calibration a bit better I might want to look for the NaN rows to see >if I can get around the problems that occurred in the original >calibration. If the data has been passed through a VOTable somewhere >in the processing, the distinction between the null and NaN rows has >been lost so I can't easily pick out the 'invalid' values. > >Sure it would be possible to use a different representation...but this >is a reasonable approach and we should be very chary of putting limits >on what data users can represent or saying how they have to represent >it in what is intended to be a generic format. I think that lack of >full support for IEEE floating point numbers is a substantial limit >for VOTables. The fact that a standard developed in the 1970's has >the same limit isn't really a good reason that we need to propagate >the constraint. > > >> >> * do you know any relational database which accepts to enter >> NaN (and/or +/-Inf) values ? As far as I know, these IEEE >> floating-point patterns are not handled (the relational databases >> I use generate arithmetic errors, and refuse the operation which >> could lead to NaN's or Inf's) >> > >It is my understanding (and personal experience in the first case) >that Postgres, Oracle and Ingres support NaNs and Infs. As far as I >can tell Sybase, MySQL and SQLServer do not. (See, e.g., 8.1.3 in >http://www.postgresql.org/docs/8.2/static/datatype-numeric.html or >http://www.techonthenet.com/oracle/functions/nanvl.php) > >I'm not clear if databases that do not allow NaNs as internal values >in databases also preclude them as calculated results, e.g., what the >result of > select sqrt(-1.) >is. > > >> And yes, the VOTable document assumes (as FITS does) that NaN >> and NULL have the same semantic meaning -- i.e. that the answer >> is "no" to both questions. >> >> Cheers, francois >> >>> >>> I'm not sure that Mark's comments really addresses the Randy's >>> question. Suppose we have an original dataset, O, in some non-VO >>> format and a VO serialization of this dataset, V. Both O and V may >>> contain NaNs. As Mark points out a NaN in V is the recommended >>> representation for a null value. So in any context in which null >>> values are distinct from NaNs, VOTables cannot distinguish them, i.e., >>> a NaN in the VOTable does not in general mean that there was a NaN in >>> the original data. So if you wish to preserve NaNs VOTables are not >>> currently a safe way to do so. >>> >>> As alluded to, this particular issue has been discussed before and >>> some thoughts have been collected at >>> http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues. >>> >>> Tom >>> >>> Mark Taylor wrote: >>>> Mike, >>>> >>>> in fact VOTable does permit NaN values in (single or double) floating >>>> point data; I think this is a consequence of the design decision >>>> that the VOTable model for data is to be as close as possible to >>>> that of FITS. Furthermore, NaN is how VOTable recommends to represent >>>> NULL values in floating point data (again, following FITS) - whether >>>> that's a good idea or not is a question that has been debated elsewhere, >>>> but that's what VOTable section 6 says >>>> (http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC41) >>>> >>>> Mark >>>> >>>> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote: >>>> >>>>> I took the question differently: If VO allows FITS data, and >>>>> FITS data allows NaN, then apps should of course allow >>>>> for this. OTOH, if the question is whether "VO data" as is >>>>> serialized in a VOTable allows NaN values the I think the >>>>> answer is 'no' (but I'd have to check). There are similar issues >>>>> with how NULL values are handled, but again it depends on >>>>> whether it is in the serialized VOTable or the end data product >>>>> being accessed. The DAL protocols don't say anything about >>>>> NaN/NULL beyond how they might be serialized, FITS is FITS >>>>> and if that's what the app retrieves in the end and then that is >>>>> the standard to follow when interpreting the data. Was that >>>>> your question? >>>>> >>>>> My $0.02, >>>>> -Mike >>>>> >>>>> >>>>> >>>>> On Mon, Sep 26, 2011 at 2:06 AM, Mark Taylorwrote: >>>>> >>>>>> On Thu, 22 Sep 2011, Randy Thompson wrote: >>>>>> >>>>>>> As a general question, does data containing NaN values >>>>>>> violate any VO standards or protocols,and if not, should VO >>>>>>> applications be expected to accept them as input? >>>>>> >>>>>> the question is rather broad ("data" can take many forms), but on >>>>>> the whole the answer is that most standards and software in the VO >>>>>> should and do behave sensibly in the presence of NaN-valued floating >>>>>> point values. >>>>>> >>>>>> -- >>>>>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>>>>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>>>>> >>>>> >>>> >>>> -- >>>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> ======================================================================= >> Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg >> 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 >> Email: francois at astro.u-strasbg.fr (France) Fax: +33-(0)368 85 24 17 >> ======================================================================= ======================================================================= Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 Email: francois at astro.u-strasbg.fr (France) Fax: +33-(0)368 85 24 17 =======================================================================