an International Virtual Observatory data exchange format

an International Virtual Observatoryan International Virtual Observatory

data exchange format data exchange format

VOTable

Roy WilliamsFrançois Ochsenbein

Clive DavenhallDaniel Durand

Pierre FerniqueDavid Giaretta

Robert HanischTom McGlynn

Alex Szalay Andreas Wicenec

XML: Structured Information

<From>Antonio Stadivarius</From><To>Domenico Scarlatti</To><Date>

<Day>13</Day><Month>4</Month><Year>1723</Year>

</Date><Body>Io bisogno una appartamento acoglienti a Cremona …</Body>

4/13/23

April 13, 1723

17.iv.1723

Separation of structure from presentation

The computer can read the document:“Find all memos from April 1723”

VOTable

• Full metadata representation• Hierarchy of RESOURCEs• containing PARAMs and TABLEs• UCD (unified content descriptor)

– a has unit meter

– a has UCD ORBIT_SIZE_SMAJ (Semi-major axis of the orbit )

• Can reference remote and/or binary streams• Table can be

– Pure XML

– "Simple Binary"

– FITS Binary Table

VOTable Parentage

• Astrores• XML format for tables• Developed at CDS Strasbourg• Presented at ADASS 1999• Vizier implementation

• XSIL• XML format for Tables and Arrays• Developed at LIGO Caltech 2000• Extensible through Type-Class dynamic loading• Java parsing, browsing, editing• Matlab interface

Sample VOTable

<?xml version="1.0"?><!DOCTYPE VOTABLE SYSTEM "http://us-vo.org/xml/VOTable.dtd"><VOTABLE version="1.0"> <DEFINITIONS> <COOSYS ID="myJ2000" equinox="2000." epoch="2000." system="eq_FK5"/> </DEFINITIONS> <RESOURCE> <PARAM name="Observer" datatype="char" arraysize="*" value="William Herschel"> <DESCRIPTION>This parameter is designed to store the observer's name </DESCRIPTION> </PARAM> <TABLE name="Stars"> <DESCRIPTION>Some bright stars</DESCRIPTION> <FIELD name="Star-Name" ucd="ID_MAIN" datatype="char" arraysize="10"/> <FIELD name="RA" ucd="POS_EQ_RA" ref="myJ2000" unit="deg" datatype="float" precision="F3" width="7"/> <FIELD name="Dec" ucd="POS_EQ_DEC" ref="myJ2000" unit="deg" datatype="float" precision="F3" width="7"/> <FIELD name="Counts" ucd="NUMBER" datatype="int" arraysize="2x3x*"/> <DATA> <TABLEDATA> <TR> <TD>Procyon</TD><TD>114.827</TD><TD>5.227</TD> <TD>4 5 3 4 3 2 1 2 3 3 5 6</TD> </TR> <TR> <TD>Vega</TD><TD>279.234</TD> <TD>38.782</TD><TD>8 7 8 6 8 6</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE></VOTABLE>

<DATA> <FITS> <STREAM href="ftp://server.com/mydata.fits" expires="2002-02-22" actuate="onRequest"/> </FITS></DATA>

Table Cell

scalar

arrays

variable length arrays

etc

etc

boolean

bit

unsignedByte

short

int

long

char

unicodeChar

float

double

floatComplex

doubleComplex

Primitives

VOTable is Flexy

• eg Table of images• UCD="JPEG_IMAGE"

datatype="unsignedByte" arraysize="*"

• eg Table of URL links• UCD=“DATA_LINK"

datatype="char" arraysize="*"

VOTable Schema (xsd)

Table Data Model

• Metadata• Class definition for Row• FIELD

– data type

– semantic type

• Data• Each Row is a list of Cells• Each Cell is an array of Primitives

– may be variable length

Table Data Layout

• All metadata first – small, complex, XML

• Class definition for table record• + params, description, etc etc

• Then data– (may be) large, remote– XML | binary | FITS

• Instantiations of table record• All records MUST have same format

Param Data Model

• Param is “Table with one cell”• Like a FIELD value• But with a “value” attribute

Primitives

• All have fixed binary length

• Same as FITS primitives

• Except Unicode

datatype Meaning FIT

S Bytes

"boolean" Logical "L" 1

"bit" Bit "X" *

"unsignedByte" Byte (0 to 255) "B" 1

"short" Short Integer "I" 2

"int" Integer "J" 4

"long" Long integer "K" 8

"char" ASCII Character "A" 1

"unicodeChar" Unicode Character 2

"float" Floating point "E" 4

"double" Double "D" 8

"floatComplex" Float Complex "C" 8

"doubleComplex" Double Complex "M" 16

Multidimensional Array Cell

• A table cell can have lots of Primitives• Example: WCS parameters are arrays

– <FIELD name=“CRVAL”datatype=“double”arraysize=“2”/>

• Example: up to 10 images, each 64x64– <FIELD name="thumbs"

datatype="unsignedByte" arraysize="64x64x10*"/>

Hierarchy

• A VOTable contains RESOURCES– RESOURCE can contain:

• TABLE• RESOURCE• etc etc

• Usage example• Many observations in the file,

– each is a RESOURCE

• Each observation is– Parameters

– Calibration table

– Raw data table

Hierarchy

• New feature: GROUP

<TABLE name=“Nutation and Aberration”> <GROUP name=“Nutation”> <FIELD name=“Longitude”/> <FIELD name=“Obliquity”/> </GROUP> <GROUP name=“Aberration”> <GROUP name=“Equinox 1950.0”> <FIELD name=“C”/> <FIELD name=“D”/> </GROUP> <GROUP name=“Equinox 1955.0”> <FIELD name=“C”/> <FIELD name=“D”/> </GROUP> </GROUP></TABLE>

Unified Content Descriptors

• UCD is a “semantic type”

PHOT.INT-MAG.B Integrated total blue magnitude ORBIT.ECCENTRICITY Orbital eccentricity STAT.MEDIAN Statistics Median Value INST.QE Detector's Quantum Efficiency

• Can be resolved by web service– to description, examples, etc

• Base + Specifiers• eg error in default right ascension• POS.EQ.RA, MAIN, ERROR

VOTable Friends

XML BinaryStreaming

Table Datacube Semantics

VOTable√ √ √ √

BinX√ √ √ √

MS Dataset √ √

HDF√ √

XDF√ √

Some “self-describing” file formats

XML Parsing

SAX: Event-BasedHandlers for StartElement, Text, EndElement, etc.

Found element BookCatalogueFound element BookFound Element TitleFound Text The Cambridge Star AtlasFound End Element Title….

Parsing

DOM: Document Object ModelReturns a tree-like Document object with data attached

BookCatalogue

BookBook

Title

ISBN Parallel Computing Works!

Cambridge Star Atlas

Title

Author

Wil Tirion

Binding to make a Parser

for(int i=0; i<table.getFieldCount(); i++){ Field field = (Field)table.getFieldAt(i); String u = field.getUcd(); if(u != null && u.equals("POS_EQ_RA_MAIN")) System.out.println("Field " + i + " is for RA"); }

From the Schema an API and library is generatedJAXBBreezeCastor

This is JAVOT (Caltech)

VOTable Software

Treeview from UK-VO

VOTable Software

VOPlot from India-VO

VOTable Software

VOTool from US-VO

VOTable software

Mirage from Bell Labs

an International Virtual Observatory data exchange format

Documents

Transcript of an International Virtual Observatory data exchange format