Re: Database-- TOTALLY CONFLICTED RESPONSE! ;-)!

[Jason Swedlow]

Hi-

OK, just so everything is out in the open:

-- My lab leads development of OMERO (http://openmicroscopy.org).  This work is entirely funded by grants from the Wellcome Trust and BBSRC.
-- We founded a start-up called Glencoe Software (http://glencoesoftware.com) to deliver commercial versions of OMERO, one of which is PerkinElmer's Columbus.  I hold equity in Glencoe Software, but receive no salary.

As such, I know abit about all this, but can be accused of conflicts in all sorts of ways.  So this is just info, plus a response to some of Johan's valuable comments.

On Thu, Jan 22, 2009 at 5:20 PM, Jean-Pierre CLAMME <[hidden email]> wrote:

Has someone got a chance to try columbus from perkin.
It's based on the omero system but I was wondering how different it was from
the basic omero software.

A bunch of points:

-- OMERO is an open source development project.  We have released OMERO-Beta3.2, and are working up to our next major release, OMERO-Beta4.  See http://trac.openmicroscopy.org.uk/omero/roadmap for more info.  If you want to see what we mean by "beta", see http://trac.openmicroscopy.org.uk/omero/wiki/OmeroIsBeta.

-- Columbus is a commercial version of OMERO-Beta2.3 (http://trac.openmicroscopy.org.uk/omero/milestone/3.0-Beta2.3), with a few tweaks, sold by PerkinElmer-- they have purchased a commercial license for this software from Glencoe Software.  The major difference between Columbus and OMERO-Beta2.3 are a few speed-ups to make viewing thumbnails in 384-well plates much faster.  A version of Columbus based on OMERO-Beta4 is currently under development.

-- Currently, open source OMERO is installed and running at > 400 sites world-wide.  The best way to talk to this community is using the OME mailing lists.  See http://www.openmicroscopy.org/site/community

-- Responding to Johan, from the other thread... we tried to release a system with a dynamic data model (the OME Server), but enabling the resulting flexibility comes with a price-- alot of complexity, which many people did not want.  In the end, we have gone for something simpler-- that's OMERO.  However, OMERO supports StructuredAnnotations, which are essentially attachments for your data.  Basically, the same kind of flexibility you have with email attachments, but with a defined namespace to enable access of your specific analysis, protocols, worksheets, etc.  See http://trac.openmicroscopy.org.uk/omero/wiki/StructuredAnnotations.  If these annotations are text based (.xls, .doc, .pdf, .ppt, .csv, .txt, .xml) they can be indexed by OMERO's search engine.  See http://trac.openmicroscopy.org.uk/omero/wiki/OmeroSearch.  Obviously, this isn't a defined data model, and isn't a complete solution for everyone's needs.

-- Johan's other point-- that accessing image data from a filesystem outperforms accessing image data in OMERO.  This depends on your specific configuration, but for most cases, is certainly true.  However, OMERO renders complex multi-dimensional image data in its server, and then sends the rendered images over a network connection to a client application (Java or web) running on your own machine.  Thus, you can have access to your multi-dimensional image data anywhere there is an internet connection.  At least in our place, the performance is only slightly slower than with our dedicated image processing systems reading directly from the filesystem, but we can get full image visualization on our laptops-- in the lab, café, or at home. As always, there's a compromise-- flexibility and performance.  With OMERO, you need a network connection, and a server.  If you want to see an example of how bad or good that performance is (depending on your opinion), see the screencast videos for OMERO.insight at http://www.openmicroscopy.org/site/videos.

-- To Jean-Pierre:  yes, we do use OMERO.  Our main production server here has ~2 TB of data, and >3.5 million frames, and ~50 users.  We don't have good stats on what others are doing, but maybe ask the ome lists.

Hope that all helps. 

Cheers,

Jason

 

Thanks,

JP

--
**************************
Wellcome Trust Centre for Gene Regulation & Expression
College of Life Sciences
MSI/WTB/JBC Complex
University of Dundee
Dow Street
Dundee  DD1 5EH
United Kingdom

phone [phone]
Intl phone:  [phone]
FAX   [phone]
email: [hidden email]

Lab Page: http://www.dundee.ac.uk/lifesciences/swedlow/
Open Microscopy Environment: http://openmicroscopy.org
**************************

[mahogny]

> -- Johan's other point-- that accessing image data from a filesystem
> outperforms accessing image data in OMERO.  This depends on your specific
> configuration, but for most cases, is certainly true.  However, OMERO
> renders complex multi-dimensional image data in its server, and then sends
> the rendered images over a network connection to a client application (Java
> or web) running on your own machine.  Thus, you can have access to your
> multi-dimensional image data anywhere there is an internet connection.  At
> least in our place, the performance is only slightly slower than with our
> dedicated image processing systems reading directly from the filesystem, but
> we can get full image visualization on our laptops-- in the lab, café, or at
> home. As always, there's a compromise-- flexibility and performance.  With
> OMERO, you need a network connection, and a server.  If you want to see an
> example of how bad or good that performance is (depending on your opinion),
> see the screencast videos for OMERO.insight at
> http://www.openmicroscopy.org/site/videos.
>  

I have found that sshfs (remote filesystem over SSH) + OST (our format)
gives about 2x-3x (subjective) speed over imserv/omero on the local
network. on longer distances the latency will probably dominate. I think
speed is degraded plenty by wasteful handling of the data but this will
be solved over time. if you have many users or a very heterogenous image
collection the performance considerations are secondary; file systems
are very bad at organizing images and have poor access control
facilities. unless you do intensive image processing your operations are
not I/O-bound anyway.

I recommend as a policy that all pictures obtained in a group should be
stored *centrally* and *with good descriptions*, never on individual
computers. a database implements this naturally, otherwise you have to
set up strict rules. it will also help converting to a database later on
if you are still uncertain. it is extremely wasteful if someone has to
redo a recording just because they have no idea where an image is or if
one exists at all.

/Johan

--
--
------------------------------------------------
Johan Henriksson
MSc Engineering
PhD student, Karolinska Institutet
http://mahogny.areta.org http://www.endrov.net