Posted by
Jason Swedlow on
URL: http://confocal-microscopy-list.275.s1.nabble.com/OME-Compliant-Specification-tp5122114p5124959.html
Hi Mark-
Thanks much for your email-- very important questions. Answering in turn:
1. Aren't we just creating yet another format? Who needs that?
No, it's not another new format-- it's been around since 2004, and was
first 'published' in 2005:
http://genomebiology.com/2005/6/5/R47It's supported by a number of commercial manufacturers:
http://www.openmicroscopy.org/site/support/about/partners(and that list is increasing rapidly)
We aim to update the underlying model for this spec at least twice per year:
http://ome-xml.org/roadmap?show=allIn this 'Compliant Specification, we haven't changed anything, aside
from our regular updates and fixes. However, we do recognize that
that the full spec is alot to work with, and have been told repeatedly
that it is too large, too comprehensive (see below), and too much to
deal with (see
http://ome-xml.org/browser/Schemas/OME/2010-04/ome.xsd),
and have been asked repeatedly to provide a recommendation on some
subset of the spec that 'should be' used.
Some time ago, after alot of discussion, we released a 'Minimal
Specification' (
http://ome-xml.org/wiki/MinimumSpecification). This
is TRULY minimal-- it's what you need to display an image (in a
technical sense), and nothing more. Is this useful? That's
debatable, but it seemed to be a necessary starting point for
adoption. The 'Compliant' specification developed after alot
experience with many thousands of files donated to the Bio-Formats
repository and submitted to the JCB DataViewer-- it's the result of
alot of practical experience with what most people seem to be doing.
2. OME-XML doesn't fit some new imaging modalities-- why not?
For sure. Any specification will necessarily be behind the cutting
edge. We welcome feedback, and update the spec at least twice/year,
more often if necessary. However, we ONLY make changes if we have
authoritative information on how to model a new imaging mode. In
other words, this takes community input and comment. We've had lots
of comments and criticism (e.g., modeling multi-line TIRF, etc.) and
always welcome more. See
http://www.openmicroscopy.org/site/communityfor ways to contribute. You are also welcome to use this list-- some
of the OME gang follow this list as well.
In short, be specific, and hit us hard-- we'll respond.
3. Validation? See
http://validator.openmicroscopy.org.uk. Also,
Bio-Formats (
http://www.openmicroscopy.org/site/products/bio-formats)
has command line validation tools. Ask if you need more info.
4. Security? Let us know specifically what you're thinking of. This
is often discussed, and the use cases for security are complex. We'd
prefer to put this security into the applications that use these file
formats, but welcome other ideas.
5. Too focused on confocal? We try to be as broad as possible, but
we are certainly behind some of the latest innovations, as it always
takes time to define what is in fact necessary to properly document a
specific mode of imaging. We ARE too focused on fluorescence-- we
don't do DIC or Pol very well. Any specific suggestions are most
welcome.
6. Why not FITS?? Well, sure. Why not DICOM? HDF5? plain old
TIFF?? or any others (there are many, and each has its proponents)?
Each of those can certainly satisfy many of the requirements
microscopy has, and in some cases (e.g., DICOM) has made explicit
attempts to support biological microscopy.
There's a very important distinction to make here-- between the vessel
that stores the data (the file itself) and the structure of the
metadata. Starting with the first release of OME-XML, we made the
strategic decision to NOT specify the file format, but to focus on the
metadata only. This decision was simply the obvious and expedient
one-- as you point out, there were then, and are now, a multitude of
ways to store image data, and they are used for different applications
for good reasons. We shouldn't get in the business of arbitrating
that issue (Tom Leher's 'National Brotherhood Week' comes to mind).
However, in OME, we believe there should be a common way of expressing
the metadata that can be used in any vessel, or even live on its own
as a separate entity. FITS uses key-value pairs and doesn't provide
the flexibility we believe is necessary. XML provides a standardised
structure that can be well-documented, is supported in all software
environments, can be transformed, versioned, and converted into other
useful structures (like databases), and can also live on its own in a
known file. I am pretty sure that OME-XML could be added to the FITS
ASCII header, if that's what you wanted to do (I'm not sure of this,
and would defer to others on this).
As an example, we actively support OME-TIFF
(
http://www.ome-xml.org/wiki/OmeTiff), which takes standard TIFF, and
adds OME-XML to the header. Why TIFF? Expediency. Most common lab
image processing programmes read TIFF, and TIFF can be used to store
single images, multiple image arrays, etc. Available libraries for
TIFF are well-optimised and openly available. OME-TIFF files can be
read by any software that reads TIFF (although it might not read the
metadata). In a similar vein, we are considering actively supporting
OME-HDF5, but there are some technical issues to consider. BTW, you
can write OME-TIFF from ImageJ using the Bio-Formats plug-in. Many
commercial software vendors now also write OME-TIFF.
Isn't this flexibility self-defeating? We don't think so. First,
it's simply unrealistic to believe that the whole microscopy and
bioimaging community will accept a single file format. There are too
many strongly held views (in most cases, legitimately), too wide an
array of applications, and much too rapid innovation. However, we do
believe a standardised metadata model can be used across at least most
of these domains, especially if we keep evolving the model and the
tools that support it.
How can live with so many different flavors of image files? That's
(relatively) simple-- Bio-Formats. It already supports a huge panoply
of file formats. If we standardized metadata, maintaining Bio-Formats
would be SOOOOOOO much easier. That's the point we were trying to
make in our article.
And if you've got a better idea, let us know. Any and all feedback welcome.
Hope that helps.
Cheers,
Jason
On Mon, May 31, 2010 at 10:31 PM, Mark Cannell <
[hidden email]> wrote:
>
> Hi Jason,
>
> With respect, isn't this also introducing yet another "non-standard" format?
> I don't think your "specification" fits some of the new imaging modalities
> and contains no validation information (to help stop data fabrication). I
> think this is a weakness of your suggested standard -let alone security
> issues.
>
> I can't help wonder why you did not consider pushing/using an existing
> _scientific image_ format, such as FITS -just look at the huge amount of
> work by the IAS to establish that scientific image data format? Since FITS
> is established, tightly controlled and can hold (as far as I know) all the
> data necessary to define machine/modality/processing why did you pick the
> 'XML bandwagon'?
>
> Please don't get me wrong, I'm not saying that your effort in trying to
> establish a standard for scientific imaging files is not worthwhile and
> needed but the specification just seems too focussed on some current
> confocal modalities...
> Many regards Mark
>
--
**************************
Wellcome Trust Centre for Gene Regulation & Expression
College of Life Sciences
MSI/WTB/JBC Complex
University of Dundee
Dow Street
Dundee DD1 5EH
United Kingdom
phone (01382) 385819
Intl phone: 44 1382 385819
FAX (01382) 388072
email:
[hidden email]
Lab Page:
http://gre.lifesci.dundee.ac.uk/staff/jason_swedlow.htmlOpen Microscopy Environment:
http://openmicroscopy.org**************************