International     Virtual     Observatory Alliance

Resource Metadata for the Virtual Observatory
Version 1.01

IVOA Proposed Recommendation 23 March 2004

This version:

http://www.ivoa.net/Documents/PR/ResMetadata/RM-20040323.html

Latest version:

http://www.ivoa.net/Documents/latest/RM.html

Previous version(s):

http://www.ivoa.net/Documents/PR/ResMetadata/RM-20040126.html
http://www.ivoa.net/Documents/WD/ResMetadata/RM-20031002.html
http://www.ivoa.net/Documents/WD/ResMetadata/RM-20030801.html
http://www.ivoa.net/Documents/WD/ResMetadata/RM-20030709.html
http://www.ivoa.net/Documents/WD/ResMetadata/RSM-20030509.html
http://www.ivoa.net/Documents/WD/ResMetadata/RSM-20030206.html
http://www.ivoa.net/Documents/WD/ResMetadata/RSM-20021011.html

Editor(s):

Robert Hanisch

Author(s):

IVOA Interoperability Working Group
NVO Metadata Working Group

Abstract

An essential capability of the Virtual Observatory is a means for describing what data and computational facilities are available where, and once identified, how to use them. The data themselves have associated metadata (e.g., FITS keywords), and similarly we require metadata about data collections and data services so that VO users can easily find information of interest. Furthermore, such metadata are needed in order to manage distributed queries efficiently; if a user is interested in finding x-ray images there is no point in querying the HST archive, for example. In this document we suggest an architecture for resource and service metadata and describe the relationship of this architecture to emerging Web Services standards. We also define an initial set of metadata concepts.

Status of this document

This is a Proposed Recommendation. The first release of this document was 7 June 2002.

This is an IVOA Proposed Recommendation made available for public review. It is appropriate to reference this document only as a recommended standard that is under review and which may be changed before it is accepted as a full recommendation. A list of current IVOA Recommendations and other technical documents can be found at http://www.ivoa.net/Documents/.

Acknowledgments

Many members of the IVOA Registry Working Group, IVOA Interoperability Working Group, AstroGrid project, and NVO Metadata Working Group have made significant contributions to this document. Contributors to this document have been partly or completely supported by the following projects and programs:

• The U.S. National Virtual Observatory project, which is funded by the National Science Foundation's Information Technology Research Program under Cooperative Agreement AST0122449 with The Johns Hopkins University.
• The UK AstroGrid project, which is funded by the Particle Physics and Astronomy Research Council.
• The Astrophysical Virtual Observatory, which is funded by the fifth framework program of the European Community for research, technological development, and demonstration activities (FP5).

Contents

• Abstract
• Status
• Acknowledgements
• 1 Introduction
• 2 Architecture
• 3 Resource metadata concepts
• 3.1 Identity metadata
• 3.2 Curation metadata
• 3.3 General content metadata
• 3.4 Collection and service content metadata
• 4 Data quality assessment
• 5 Service metadata concepts
• 5.1 Interface metadata
• 5.2 Capabilities metadata
• 6 Example
• 7 Changes from previous versions

1 Introduction

An essential capability of the Virtual Observatory is a means for describing what data and computational facilities are available where, and once identified, how to use them. The data themselves have associated metadata (e.g., FITS keywords), and similarly we require metadata about data collections and data services so that VO users can easily find information of interest. Furthermore, such metadata are needed in order to manage distributed queries efficiently; if a user is interested in finding x-ray images there is no point in querying the HST archive, for example. In this document we suggest an architecture for resource and service metadata and describe the relationship of this architecture to emerging Web Services standards. We also define an initial set of metadata concepts.

2 Architecture

In order to make it easy for astronomy information services to participate in the VO, we propose a hierarchical system for metadata management. At the top level we require a minimum amount of information, sufficient primarily to note the existence of a resource and to describe who is responsible for it. At lower levels, the metadata are more extensive and complex, allowing for the description of query syntax, access protocols, and usage policies.

A resource is a general term referring to a VO element that can be described in terms of who curates or maintains it and which can be given a name and a unique identifier. Just about anything can be a resource: it can be an abstract idea, such as sky coverage or an instrumental setup, or it can be fairly concrete, like an organization or a data collection. This definition is consistent with its use in the general Web community as "anything that has an identity" (Berners-Lee 1998, IETF RFC2396). We expand on this definition by saying that it is also describable.

An organization is specific type of resource that brings people together to pursue participation in VO applications. Organizations can be hierarchical and range greatly in size and scope. At a high level, an organization could be a university, observatory, or government agency. At a finer level, it could be a specific scientific project, space mission, or individual researcher. A provider is an organization that makes data and/or services available to users over the network.

A service is any VO resource that can be invoked by the user to perform some action on their behalf.  Associated with any service is descriptive metadata about the service.  Metadata generally include information the user needs to determine if a service is of interest and how the service may be invoked.  Specific types of metadata are described below.  Note that the service itself need not be aware of the metadata that describe it.

A query service supports a query/response protocol.  The user submits a query to the service that may define characteristics of interest, and the service returns a set of information to the user.  The query may be null, e.g., a current-time service may only support a null query, and some services may respond to a null query with appropriate default actions. Non-query services may also exist, e.g., services to copy or delete files on remote files systems, to mail information to other users, to kill existing jobs, to authorize actions, etc.

A registry is a query service for which the response is a structured description of resources.  The resources described by a registry may be of any type.  The registry may support a query that allows the user to indicate which resources might be of interest.

In our model, the hierarchy of resources is one in terms of management and curation. For example, an organization may manage a collection of one or more services and even smaller organizations or projects. For example, MAST, HEASARC, IRSA, NED et al. are all resources. Each of these manages other resources, e.g., the HST archive in MAST.   They also support specific services (which are also resources) such as an HST observation log query service or a cone search service. One could in principle describe all of NASA astrophysics data holdings as a resource, or all of NVO as a resource, but aggregates of this scale circumvent the goal of being able to locate the specific resources and services of interest for a particular application.

All resources are described by metadata.  Resource metadata are generic, high-level, and independent of any specific service.  Resource metadata include

• Identity metadata, which gives the resource a name and an identifier,
• Curation metadata, which describe who supports the resource and its availability (i.e., version, release date), and
• Content metadata, which describe what kind of information is available (types of data, sky coverage, spectral coverage, etc.). Content metadata can be either general, applying to all resources, or associated more specifically with data collections and the services that deliver data from them.

Resource metadata are typically not queryable parameters in the underlying services, but rather they encompass information that now is simply "known" to users, or must be discovered through other means. Astronomers know that the HST archive includes optical images and spectra, for example, or that Vizier provides access to catalogs and tables. Resource metadata constitute a "yellow pages" of astronomical information. Resource metadata are analogous to the UDDI (Universal Description, Discovery and Integration) Web Service, and are analogous to the high-level descriptions included in the CDS GLU.

Organizations, data collections, and services can be considered as classes of resources that may each require additional metadata to fully describe it, but which are not shared by other classes. For example, a service description would need to include its inputs, outputs, and how it can be accessed. Service metadata, therefore, can be thought of as an extension of the general resource metadata: where as the resource metadata, through its content metadata, describes what is available, the service metadata describes how to access it.

Resource metadata will be collected through resource registration services, e.g., web forms that present a resource curator with the requisite fields and enumerated lists, and construct a resource descriptor in a standard format (such as VOTable). The resource registration service should not allow fields to be left unspecified. Some metadata elements may be irrelevant, unknown, or not provided by the publisher of a resource. Since "irrelevant" conveys different information than "not provided", we will adopt standard representations of these conditions:

Various applications based on the registry may choose to include or exclude certain resources based on these attributes. If a metadata element is "Not Provided" the application should make no assumption regarding applicability or relevance.

Similarly, some resources may provide quite large aggregations or collections, covering many bandpasses, types, or formats. It may be prohibitive to list all such options. In such cases acceptable representations for the metadata entries would be:

The most general resource metadata is similar in concept to the Dublin Core metadata definitions (http://dublincore.org/documents/dces/), and where possible DC metadata elements have been used. VO metadata elements that correspond directly to DC counterparts are noted. The Dublin Core elements Language and Relation are not currently used in the VO metadata.

3 Resource metadata concepts

Below we describe the concepts we believe are needed in the resource metadata. These concepts may be instantiated in a variety of standard forms, e.g. XML, UCD tags, or FITS keywords, and with a variety of mechanisms, such as Topic Maps, OWL, or RDBMSs. Consequently, the exact names and rendering of the values may depend on the particular form in which they are represented.  For example, when Coverage.Spatial is rendered as a FITS keyword record, the name will need to be limited to 8 characters and the value rendered in a pure ASCII form; in contrast, when rendered in XML, it might be better to tag the different components of the value separately.  It will be necessary to define standard renderings for each of these common forms.

A limited number of keywords are considered essential for a basic understanding of the resource, and are thus denoted as required. All others are optional, or may be applied to certain classes of resources only.

3.1 Identity metadata

3.2 Curation metadata

3.3 General content metadata

This list is extensible. Resources providing more than one type of content should list all relevant types.

Collection and service content metadata

Other format values will be used to describe the physical medium of the information: CDROM, Digital Planetarium, Online, Presentation, Print, Slides, Video. Format specifications may be combined, as in "Video, video/mpeg" (both hardcopy video cassettes and on-line MPEG files) or "CDROM, image/fits, image/gif" (FITS and GIF images are available on-line and on CDROM hardcopy).

Data quality assessment

Users of virtual observatory resources need some way to assess the quality of the data. Data quality is both subjective and quantitative, and data collections may have no single data quality metric. While the completeness and consistency of the resource metadata itself may be a reasonable indicator of the quality of the associated resource, this is at best a qualitative measure. The following metadata elements are intended to capture the most basic measures of data quality, and may well require extensions as VO usage practices evolve and become more sophisticated.

Service metadata concepts

The metadata necessary for describing a service will vary quite a bit depending on the type of service it is. We propose two general categories of service metadata:

Interface metadata, which describe how to access the service-the inputs and the outputs. There will be standard types of interfaces that could include a web-browser-based interface (i.e., HTML Forms), a Web Service interface (describable by a WSDL document), a general HTTP Get interface (e.g., using key=value arguments), and a GLU-described interface.

Capability metadata, which describe what the service does, its limitations, and other behavioral characteristics.

Note that these categories are reasonably orthogonal. We can imagine the same basic service-in terms of its capabilities-accessible through multiple interfaces.

We expect that for each standard service recognized by the VO there will be a specification document that defines all the specific metadata necessary to describe a particular implementation of that service; thus, we do not include them all here.  However, we can identify a few metadata concepts that might be employed to describe a particular service.  Described below, these concepts should be employed by standard service specifications wherever they are applicable. We note also that metadata associated with the VOTable schema can also be reused to describe the inputs and outputs of a service that returns a VOTable.

Interface metadata

Capabilities metadata

Example

Example: The Sloan Digital Sky Survey data as hosted by MAST at STScI (with no assertion that the metadata element values are actually correct, though they are not unreasonable).

Changes from previous versions

From V1.0 to V1.01

• Revised wording in Section 2 to clarify phrase concerning "independence" of resource metadata.
• Added text to indicate that PublisherID must have the form of a valid IVOA Identifier (Section 3.2).
• Clarified definition of ReferenceURL (Section 3.3).
• Added text to indicate that RelationshipID must have the form of a valid IVOA Identifier (Section 3.3).
• Deleted Convex from list of possible values for Spatial.Coverage, as it is redundant with extant ability to specify region intersections (Section 3.4).
• Corrected URL to coordinate system descriptions at CDS ( http://aladin.u-strasbg.fr/java/doctech/cds/astro/Astroframe.html, Section 3.4).
• Renamed Coverage.Spectral value of UV to Ultraviolet, and deleted the value EUV. EUV spectral range is merged into UV. (Section 3.4.)
• Renamed Service.HTTPResults to Service.HTTPResultsMIMEType (Section 5.1 and in example).
• Renamed Service.StandardURI to Service.StandardID, consistent with example and with other elements that have the form of URIs/Identifiers. This was an inconsistency not noticed by reviewers previously. Also added text to indicate that Service.StandardID must have the form of a valid IVOA Identifier. (Section 5.2)
• Corrected URL in the example for element Service.InterfaceURL.

From V0.82 to V1.0

• Removed Creator from list of required elements (Section 3.2).
• Removed Contact.Address and Contact.Telephone, and modified description of Contact to reflect the removal of these elements (Section 3.2).
• Updated the example in Section 6 to reflect the above changes.

From V0.81 to V0.82:

• Included full change history from previous versions.
• Identified a subset of metadata elements as required for all resources. These are Title, Identifier, Publisher, Creator, Subject, Description, ReferenceURL, and Type.
• Added reference to IVOA Identifiers specification under metadata element Identifier (Section 3.1).
• Added Contact.Address and Contact.Telephone, and modified description of Contact to reflect these new elements (Section 3.2).

• Added alternate sources of Subject keywords in addition to the IAU Thesaurus (Section 3.3).
• Clarified definition of Source to avoid confusion with Dublin Core sense of "resource" (Section 3.3).

• Added Registry to list of recognized resource Types (Section 3.3).
• Added Other to list of recognized resource Types (Section 3.3).
• Added Relationship and RelationshipID to provide means to show relationships (mirror of, service for, derived from) between resources (Section 3.3).
• Restructured Section 3.3 to distinguish the most generic content metadata (which remains in Section 3.3) from content metadata that applies more specifically to data collections and their corresponding data access services (new Section 3.4).
• Refined definition of Coverage.Spatial (Section 3.4).

• Inserted Millimeter bandpass definition into Coverage.Spectral (Section 3.4).
• Added UCD to Section 3.4 to provide a way to show the contents of resources via their associated Unified Content Descriptors.
• Updated the example in Section 6 to include new metadata elements and use Identifiers that are consistent with the most recent drafts of the Identifier specification.
• Added acknowledgements to funding agencies.

From V0.8 to V0.81:

• Restricted units of Coverage.Depth to Jy and eliminated Coverage.Depth.Units. It is assumed that a registry data entry interface could handle unit conversions.
• Added section on data quality metadata, including elements DataQuality, Uncertainty.Photometric, Uncertainty.Spatial, Uncertainty.Spectral, and Uncertainty.Temporal.
• Changed names of Service-related metadata elements to separate Service root from the rest of the name (e.g., ServiceInterfaceURL → Service.InterfaceURL).
• Changed name of ServiceMSR to Service.MaxSearchRadius.
• Added Service.MaxReturnRecords.
• Updated example to reflect above.

From V0.7 to V0.8:

• Title: Document renamed to "Resource Metadata for the Virtual Observatory" recognizing that services are a subset of resources.
• Section 2: Made several minor wording changes to reflect generic use of the term "resource", which encompasses "service".
• Section 2: Made definition of "Curation Metadata" more specific.
• Section 2: Added definitions for metadata elements that are not provided, unknown, or not applicable. Replaces previous discussion at end of Section 3.3.
• Section 2: Added definitions for aggregate resources whose contents might include "any" or "all" of a given metadata element's types.
• Sections 2 and 3: Added note in Section 2 describing relationship of VO metadata to Dublin Core. In Section 3, changed references to Dublin Core to cite those elements that are the same as DC rather than those elements that are different from DC.
• Section 3.1: Changed Ticker to ShortName and extended string length from 8 to 16 characters.
• Section 3.2: Changed "based on" to "drawn from" the IAU Astronomy Thesaurus, concerning the selection of Subject metadata.
• Section 3.2: Moved elements Title, Subject, and ReferenceURL from 3.2 to 3.3.
• Section 3.3: Added metadata element Source (from Dublin Core), allowing for a cross reference to the bibcode (or other form of citation) for an associated bibliographic reference.
• Section 3.3: Extended Type to include Organization and Project.
• Section 3.3: Modified Coverage.Spatial to conform to Region specification of Space-Time Metadata definitions.
• Section 3.3: Clarified definition of RegionOfRegard (which is NOT the angular resolution).
• Section 3.3: Added Coverage.Depth.Units to provide separate field for units string.
• Section 3.3: Added Coverage.SkyFraction.
• Section 3.3: Added Resolution.Spatial, Resolution.Spectral, and Resolution.Temporal elements.
• Section 3.3: Allowed Facility and Instrument elements to be lists.
• Section 3.3: Changed examples for Format to use standard MIME types.
• Section 5: Updated example to include above changes and additions.

From V0.6 to V0.7:

• Reformatted document to be compatible with new IVOA documentation standards.
• Modified much of the architecture description to reflect current concepts about the function and structure of registries.
• Distinguished identity metadata from other curation metadata.
• Added Ticker element to provide shorthand abbreviation for resource Title.
• Added PublisherID element to provide unique identifier for the resource publisher.
• Replaced ResourceURL with ReferenceURL.
• Deleted ServiceURL and incorporated more general service metadata (Section 4).
• Incorporated more complete metadata definitions for Education and Public Outreach resources. Affects: Type, Format, ContentLevel.
• Added Type = Simulation for theoretical models.
• Coverage.Spatial refers to Space-Time Metadata document for spatial coverage specifications.
• Added Coverage.Spectral.CentralWavelength, Coverage.Spectral.MinimumWavelength, and Coverage.Spectral.MaximumWavelength to more fully describe the spectral bandpass.
• Replaced Coverage.Temporal with Coverage.Temporal.StartTime and Coverage.Temporal.StopTime to be consistent with Date and use of ISO 8601 time format.
• Added Coverage.RegionOfRegard to aid users of the registry in assessing the utility of a resource.
• Added Coverage. Depth, Coverage.Object Density, and Coverage.ObjectCount to provide information on resource depth of coverage (sensitivity) and richness.
• Broadened definition of Format, consistent with Dublin Core, to include specifications of both physical and digital formats.
• Added new section on service metadata, introducing metadata elements ServiceInterfaceURL, ServiceBaseURL, ServiceHTTPResults, ServiceStandardID, ServiceStandardURL, and ServiceMSR.
• Updated example to use new metadata elements.

From V0.5 to V0.6:

• Added Date element, following Dublin Core and in response to October 2002 discussion.
• Added Version element, in response to October 2002 discussion. Not in Dublin Core.
• Added Creator.Logo element, in response to October 2002 discussion. Not Dublin Core.
• Added Subject element, to replace Coverage.Topics, as the former is Dublin Core, the latter is not, and the intent is basically the same.
• Added ContentLevel element, to replace Coverage.Level. Upon reflection I thought that ContentLevel was both a better label and was of a different nature than the other Coverage elements. M. Voit will provide further inputs on appropriate element values.
• Updated example to use new metadata elements.