skoda at sunstel.asu.cas.cz
Mon Jun 29 17:51:39 PDT 2009
before I finished the ideas, your comments came - so I will just repeat
in other words my feelings:
> especially in the case of the small, older collections we are mainly
> talking about here. Someone would have to be willing to this work.
I know several people who are eager but it has to be relatively easy
>> By the way, I don't think the FITS format described in the SDM was
>> selected because it was a common FITS format.
No, I think as well it is not so common (except satellites)
It was choosen because
>> it was a logical format for storing and describing,
Yes it is a natural model for general spectrum
>> and in theory, could be expanded to support SEDs, light curves,
>> echelle spectra, etc.
sure - the DAL effort is based on this idea and the SDM
will probably become the base for general data set description (am I right
> The binary table FITS format specified by SDM is fairly widely used,
In some specific domain - to be honest I haven't met yet in my list of
optical ground based collections the bintable fits as the calibrated
scientific result (probably partly due to the simple reason:
The most of ground optical astronomers do not know what it is exactly
bintable FITS and they would not be able to display it ;-)
No joking - my personal survey among colleagues from variable star
research - some are aware of it however - (answer of one colleague who
works with Galex, EUVE and SDSS data : "it is a strange kind of FITS,
(not the proper one ;-) used by satellites - the only way you can do with
it is to use IRAF TABLES tcopy to print to ascii table and you can use
rspectext to convert in proper (1D FITS) IRAF file to work with splot"
other just came to me - I have spectra in FITS but it is corrupted - IRAF
will not show it !
> (as is VOTable of course). Using a simple 1D image for spectra
> on the other hand has serious limitations, e.g., one cannot have
> an associated error vector, there is no way to deal with variable
> resolution, a background vector cannot be provided, etc.
Absolutely - it is a one of problems with echelle reduction - the merging
of orders and rebining in wavelenght is incorrect and introduces errors !
All the effort of UVES, HARPS, FEROS, HEROS .... pipelines is in vain
converted to this format. However, it is the final science result and
people want it ! Very few people care about sigma errors . They want to
see the spectrum shape, the background should be substracted already.
I am afraid that it would be nice if the astronomers were educated by VO
standards to properly work with spectra (and realize the issues of
background, sigma errors etc - but even IRAF has the support for this for
very short time an only in specific packages (e.g. gemini), not in common
ones recommended in tutorials...
Not talking about MIDAS FITS convertor (from MIDAS tables) ...
> says this use of a 1D image probably originated with IRAF, but IRAF
> moved on to more complex formats such as multispec format
yes the result of reduction is multispec with 4 bands - but many
reductions are finished by scopy to onedspec or the extensions are
> bintable for many IRAF-based projects
- well all space based AND perhaps Gemini
> Due to the serious limitations of a 1D FITS image as a spectral format
> I don't think we should encourage this in the VO era.
Not encourage but tollerate it in clients (maybe in some transition period
before the new SDM and SxAP for GDS (general data sets) is approved.
But we have to be able to show some substantial amount of SSA services
with various set of spectra to be consumed in ALL VO clients...
I am still doubting about the sense of current spectra services (except
the role of visualisation - the serious analysis still has a lot of
problems including the limitation of SDM as implementes...
> supported anyway already via native pass-through in a "query compliant"
Astronomers do want to get a list of spectra - they want to see them in VO
> issue is client code then this could be supported as an optional
> output format; CSV/TSV for example is already also supported for
> simple clients.
I do not understand the difference in supported as optional output format
(case of CSV),
and the explicit list in table in chapter 22.214.171.124 od SSAP 1.04.
> How many of these small spectral archives are really worth trying
> to get into the VO, given issues such as questionable or nonexistent
The key question is - what is proper VO metadata ? The full char,
provenance and curation, all STC ?
Show me service supporting all of this !
I bet 99.999 of spectra comming from SSA services do not have proper
metadata (including the latest ESO SSA) I do not know about space project
what it is required.
But I can tell all the observatories producing the spectra have enough
metadata described in FITS headers all elsewhere in Logs, to be able to do
their science analysis. And I am afraid we will ever have the whole
provenance and characterisation in all data sets as required bymodels in
preparation. It would require formal description of whole pipeline, all
information from telescope (including the meteo - e.g. the humidity
influencing telluric lines....) and much more else (Igor could list for
several more pages .....)
And still not talking about IFU spectra, multigrating and multichip
echelles etc ...
But despite this lack of metadata - one look at spectrum shape will tell
to experienced astronomer more than reading pages of metadata .... ;-)
>If there are enough such cases where it would be worthwhile,
every information counts ! We want to pretend that VO is a heritage of
mankind where the new kind of research is allowed by the technology.
And that implies that ALL of the astronomical knowledge should be there.
Do we more care about proper curation and provenance of Almagest tables or
the content ?
What is the border between VO as presented to getting funds (unique,
federalization, all archives .... bridging the Digital divide) and the VO
as the just another version of large (space) mission archives - like
And even with the space missions - will we have Spitzer, Chandra, Corrot,
Kepler, Herschel, Planck data in VO ? In what format when the consortia
have been developing different non-VO compatible archives ?
> I might agree that some step should be taken. But I still do not
> personally know of many such cases, at least from the major ground
> based observatories.
> Are these groups even operating a basic archive
> yet where such spectra are available?
Some kind of it for internal purposes based on some public RDBMS
(postgresql, Mysql) and Apache is quite common but the data are here as it
is. Those who are not showing data might be convinced by promissing them
the VO access will make their facility more famous ....
An still there is a way of using VO tools (if enough enhanced for
particular type of analysis) for the work with proprietary data in local
environment instead of legacy tools.
> likely to solve the problem. We need a broader inventory of sites
> which have such data and which are willing and able to do at least
> some work to publish their spectra.
To convince some of them, a clear advantage of VO spectra access has to be
demonstrated - and after my experience with SSA servers and spec tools it
is still not massive scaled science (hardly several spectra can be handled
at once, not hundreds.... And the lack of "prenormalisation" and
basention of absolute flux calibration is critical
to display different spectra in one image. (I am preparing some critical
review of SSA from the point of view of a stellar astronomer).
So again the VO client development is important ..
And still there is a lot of spectra in ascii tables in Vizier that should
be SSA accessed - I am not sure about the TAP but it seems not support
spectra in its current version...
One example for all:
here is the real science content - even if lacking some metadata.
> If there were enough such sites to warrant some solution then I still
> think giving them a ready to use framework ("SSA in a box") that can
> directly ingest the FITS files and handle all the intricacies of SSA
> might be a simpler solution for a small site with limited resources
Yes I have noted this in previous email - e.g SAADA is near this goal.
> might be to upload the spectra to a spectral archive service, in
> which case the smaller site does not even have to operate an archive.
have a look at the BeSS and its content (if it is still accesible)
> In summary a more concrete use case is needed. If all we are trying
> to do here is help ESO and SAO
But once this is solved at ESO ... there will be the example of correctly
prepared spectra archive that may be used as study case for the smaller
> small data provider then there are other important issues which would
> have to be considered as well.
Exactly ! I tried to put some issues for consideration.
Now I am done with all the experience and feelings of small data providers.
This is now turn of Alberto as a representant of the richest and most
powerful astronomical company in the ground based instrumentation ;-)
* Petr Skoda Phone : +420-323-649201, ext. 361 *
* Stellar Department +420-323-620361 *
* Astronomical Institute AS CR Fax : +420-323-620250 *
* 251 65 Ondrejov e-mail: skoda at sunstel.asu.cas.cz *
* Czech Republic *
More information about the dm