analyzing really big data sets

[Watkins, Simon C-2]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Folks, Data sets are getting bigger and bigger... many hundreds of gigabytes commonly.  This is particularly pressing with light sheet data or 3D imaging of cleared tissue samples.
My question for the forum is how are you dealing with this? Many of the common packages we all use support CUDA based rendering and processing but do any allow multi-CPU processing or a distributed architecture.
How about hardware solutions?
I am not so much interested in how data is being moved with this post, that really another question and I think more readily solved.  I would like to know specifically about quantifying and rendering massive multicolor, 3D data sets
If commercial suppliers wish to contact me offline my direct mail address is below.

Simon Watkins Ph.D
Distinguished Professor and Vice Chair Cell Biology
Professor Immunology
Director Center for Biologic Imaging
University of Pittsburgh
Bsts 225 3550 terrace st
Pittsburgh PA 15261
[hidden email]<mailto:[hidden email]>
Www.cbi.pitt.edu<http://Www.cbi.pitt.edu>
412-352-2277

[George McNamara]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Simon,

Processing for deconvolution: both Microvolution.com and SVI Huygens
support NVidia GPU (microvolution can support more than one, haven't
checked for Huygens). Latest NVidia GeForce GTX 1070 and 1080 cards
require a power cable from the power supply. NVidia just announced new
Titan X (hopefully no power cable needed).

For digital slide imaging (IHC, few and more color immunofluorescence) I
am starting to use Definiens and have high hopes for Visiopharm.

AMIRA 3D software is now part of FEI (which I now think of a the cryo-EM
company) which is being acquired by ThermoFisher. I doubt this means
that every Floid, Evos, or Cellomics will ship next year with AMIRA (if
they did, the big four microscope companies and traditional image
analysis companies might then take a hit).

I still do not understand CellProfiler. Fiji ImageJ2 is not doing much
for me. MetaMorph is showing its age and failed to move into 64-bit
scale data (limit of 32000x32000 pixel image size, 65000 objects in
Integrated Morphometry Analysis).

Multiplex immunofluorescence and/or FISH: Garry Nolan and colleagues
should be close (this year) to publishing MIBI-TOF: 40plex, 20x20x~20 nm
voxel size, 1 mm^2 in 30 minutes (Garry mentions in his talks that one
mAb + "CyTOF" detction polymer:heavy metal is about this volume, so one
voxel = one molecule). Think imaging CyTOF. On the fluorescence side, I
am looking forward to their CODIS (something like George Church FISSEQ,
Xiaowei Zhuang, Cai Long have published similar ... and maybe CODIS and
Expansion Microscopy will work well together). ... I can also see modest
plex fluorescence and then comprehensive single cell RNAseq, something
along the lines of Tirosh 2016 (PubMed 27124452). Some of my thoughts
about fluorescence multiplexing to 20plex and beyond (I've not replaced
eight with ten since hearing about BD FACsymphony with decagons):

https://www.linkedin.com/pulse/bd-biosciences-listed-tandems-horizon-brilliant-violets-mcnamara


big data analysis:

TIBCO Spotfire

looks promising.

lighter weight and/or harder to use: RapidMiner, Origin Pro (9 million
cells vs max 1 million rows for Excel), Tableau for visualization and
Tableau for Student(s) is free for one year - just keep finding a new
student every year per workstation.

For my consulting (between sending job applications) I now use a
Yamasaki 40" UHD 3840 x 2160 pixel monitor ($500) for digital slide
image analysis. Internet connectivity is pretty good (comcast at home),
remote desktop connection to the remote workstation.

https://www.amazon.com/YAMAKASI-O40USUT-Inch-Monitor-10Bit/dp/B017VWX3U8
YAMAKASI O40USUT 40 Inch UHD / 4K Monitor (3840 x 2160) HDMI 2.0, UHD
(60Hz), HDCP 2.2, MHL, 10Bit
I discovered online how to switch the display from Korean to English.
HDMi2.0->DisplayPort cable arrives today since standard HDMI1.x cable
limits the monitor to 30Hz. I also found online that HDMI #4 is the only
port on the monitor with 60 Hz capability (if HDMI2.0 cable and assuming
graphics card works). 40" is a bit too big to easily use two on one system.

//

In addition to the montly (weekly?) new papers on Light Sheet, I am
excited about the Expansion Microscopy. See Ed Boyden's papers 88 and 89
downloadable at
http://syntheticneurobiology.org/publications

plus opportunities to combine ExM with light sheet.

George


On 7/26/2016 10:26 AM, Watkins, Simon C wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Folks, Data sets are getting bigger and bigger... many hundreds of gigabytes commonly.  This is particularly pressing with light sheet data or 3D imaging of cleared tissue samples.
> My question for the forum is how are you dealing with this? Many of the common packages we all use support CUDA based rendering and processing but do any allow multi-CPU processing or a distributed architecture.
> How about hardware solutions?
> I am not so much interested in how data is being moved with this post, that really another question and I think more readily solved.  I would like to know specifically about quantifying and rendering massive multicolor, 3D data sets
> If commercial suppliers wish to contact me offline my direct mail address is below.
>
> Simon Watkins Ph.D
> Distinguished Professor and Vice Chair Cell Biology
> Professor Immunology
> Director Center for Biologic Imaging
> University of Pittsburgh
> Bsts 225 3550 terrace st
> Pittsburgh PA 15261
> [hidden email]<mailto:[hidden email]>
> Www.cbi.pitt.edu<http://Www.cbi.pitt.edu>
> 412-352-2277

--


George McNamara, PhD
Houston, TX 77054
[hidden email]
https://www.linkedin.com/in/georgemcnamara
https://works.bepress.com/gmcnamara/75/

[Tim Feinstein]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Has anyone used the workflow from Amat...Keller (2015) in Nature
Protocols?  They report (Fig 5) that their new compression/analysis
toolkit works well with multiple processor cores, especially compared with
other compression alternatives like LZW or JPEG2000.  That may be a good
way to make the most of distributed computing resources when those are
available.  

Expansion microscopy (effectively) increases resolution but cuts working
distance.  Does it help to have larger structures if you need to look
deeper to see them?

Best,


Tim

Timothy Feinstein, Ph.D.
Research Scientist
University of Pittsburgh Department of Developmental Biology





On 7/26/16, 12:45 PM, "Confocal Microscopy List on behalf of George
McNamara" <[hidden email] on behalf of
[hidden email]> wrote:

>*****
>To join, leave or search the confocal microscopy listserv, go to:
>https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.
>edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7ctnf8%40PITT.ED
>U%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c
>1&sdata=zCFGxcxeouEzk9bGXxtIR3FZ1aUPhBF8q5JmT9rrif0%3d
>Post images on
>https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.
>com&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9e
>f9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=6olqkfmArvSQrGctzPKRG0WQbssJUEK8Z
>ZLdukDXTzA%3d and include the link in your posting.
>*****
>
>Hi Simon,
>
>Processing for deconvolution: both Microvolution.com and SVI Huygens
>support NVidia GPU (microvolution can support more than one, haven't
>checked for Huygens). Latest NVidia GeForce GTX 1070 and 1080 cards
>require a power cable from the power supply. NVidia just announced new
>Titan X (hopefully no power cable needed).
>
>For digital slide imaging (IHC, few and more color immunofluorescence) I
>am starting to use Definiens and have high hopes for Visiopharm.
>
>AMIRA 3D software is now part of FEI (which I now think of a the cryo-EM
>company) which is being acquired by ThermoFisher. I doubt this means
>that every Floid, Evos, or Cellomics will ship next year with AMIRA (if
>they did, the big four microscope companies and traditional image
>analysis companies might then take a hit).
>
>I still do not understand CellProfiler. Fiji ImageJ2 is not doing much
>for me. MetaMorph is showing its age and failed to move into 64-bit
>scale data (limit of 32000x32000 pixel image size, 65000 objects in
>Integrated Morphometry Analysis).
>
>Multiplex immunofluorescence and/or FISH: Garry Nolan and colleagues
>should be close (this year) to publishing MIBI-TOF: 40plex, 20x20x~20 nm
>voxel size, 1 mm^2 in 30 minutes (Garry mentions in his talks that one
>mAb + "CyTOF" detction polymer:heavy metal is about this volume, so one
>voxel = one molecule). Think imaging CyTOF. On the fluorescence side, I
>am looking forward to their CODIS (something like George Church FISSEQ,
>Xiaowei Zhuang, Cai Long have published similar ... and maybe CODIS and
>Expansion Microscopy will work well together). ... I can also see modest
>plex fluorescence and then comprehensive single cell RNAseq, something
>along the lines of Tirosh 2016 (PubMed 27124452). Some of my thoughts
>about fluorescence multiplexing to 20plex and beyond (I've not replaced
>eight with ten since hearing about BD FACsymphony with decagons):
>
>https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.linke
>din.com%2fpulse%2fbd-biosciences-listed-tandems-horizon-brilliant-violets-
>mcnamara&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6
>%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=wYGfzZEok0RA5eI3MNyLA2P%2bC4
>mJwog2xCHuxpWISOY%3d
>
>
>big data analysis:
>
>TIBCO Spotfire
>
>looks promising.
>
>lighter weight and/or harder to use: RapidMiner, Origin Pro (9 million
>cells vs max 1 million rows for Excel), Tableau for visualization and
>Tableau for Student(s) is free for one year - just keep finding a new
>student every year per workstation.
>
>For my consulting (between sending job applications) I now use a
>Yamasaki 40" UHD 3840 x 2160 pixel monitor ($500) for digital slide
>image analysis. Internet connectivity is pretty good (comcast at home),
>remote desktop connection to the remote workstation.
>
>https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.amazo
>n.com%2fYAMAKASI-O40USUT-Inch-Monitor-10Bit%2fdp%2fB017VWX3U8&data=01%7c01
>%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87
>cc3a526112fd0d%7c1&sdata=%2fBPJHQdoQQN6V6kckROHCV2l4xBshMEp4kD2znPP9p8%3d
>YAMAKASI O40USUT 40 Inch UHD / 4K Monitor (3840 x 2160) HDMI 2.0, UHD
>(60Hz), HDCP 2.2, MHL, 10Bit
>I discovered online how to switch the display from Korean to English.
>HDMi2.0->DisplayPort cable arrives today since standard HDMI1.x cable
>limits the monitor to 30Hz. I also found online that HDMI #4 is the only
>port on the monitor with 60 Hz capability (if HDMI2.0 cable and assuming
>graphics card works). 40" is a bit too big to easily use two on one
>system.
>
>//
>
>In addition to the montly (weekly?) new papers on Light Sheet, I am
>excited about the Expansion Microscopy. See Ed Boyden's papers 88 and 89
>downloadable at
>https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fsyntheticn
>eurobiology.org%2fpublications&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507
>a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=RweqB9
>SKyLpFwTGyjkokHEkOIjXL6Ih79VVZJRK%2f0Eo%3d
>
>plus opportunities to combine ExM with light sheet.
>
>George
>
>
>On 7/26/2016 10:26 AM, Watkins, Simon C wrote:
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>>
>>https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn
>>.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7ctnf8%40PITT.
>>EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d
>>%7c1&sdata=zCFGxcxeouEzk9bGXxtIR3FZ1aUPhBF8q5JmT9rrif0%3d
>> Post images on
>>https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur
>>.com&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c
>>9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=6olqkfmArvSQrGctzPKRG0WQbssJUE
>>K8ZZLdukDXTzA%3d and include the link in your posting.
>> *****
>>
>> Folks, Data sets are getting bigger and bigger... many hundreds of
>>gigabytes commonly.  This is particularly pressing with light sheet data
>>or 3D imaging of cleared tissue samples.
>> My question for the forum is how are you dealing with this? Many of the
>>common packages we all use support CUDA based rendering and processing
>>but do any allow multi-CPU processing or a distributed architecture.
>> How about hardware solutions?
>> I am not so much interested in how data is being moved with this post,
>>that really another question and I think more readily solved.  I would
>>like to know specifically about quantifying and rendering massive
>>multicolor, 3D data sets
>> If commercial suppliers wish to contact me offline my direct mail
>>address is below.
>>
>> Simon Watkins Ph.D
>> Distinguished Professor and Vice Chair Cell Biology
>> Professor Immunology
>> Director Center for Biologic Imaging
>> University of Pittsburgh
>> Bsts 225 3550 terrace st
>> Pittsburgh PA 15261
>> [hidden email]<mailto:[hidden email]>
>>
>>https://na01.safelinks.protection.outlook.com/?url=Www.cbi.pitt.edu&data=
>>01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0
>>a04eeb87cc3a526112fd0d%7c1&sdata=EwPWpth3tusWtzmY5Er6JXMPi2c1IL9QvYM9hqQI
>>rfM%3d<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fWw
>>w.cbi.pitt.edu&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b
>>5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=ejZynFRdwaWxJIESmghC
>>SUX1iDF4tqXYqrtU4XQ2o1s%3d>
>> 412-352-2277
>
>--
>
>
>George McNamara, PhD
>Houston, TX 77054
>[hidden email]
>https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.linke
>din.com%2fin%2fgeorgemcnamara&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a
>94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=9TyvMQA
>YK7XWLKn3vHttEg1q%2bN%2fAHYgEG9T2xiN1lYc%3d
>https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fworks.bep
>ress.com%2fgmcnamara%2f75%2f&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a9
>4cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=Ruc%2fyM
>5MvOXiUpqTtIXXULVMyHSyb4U5Y2DsR278MhA%3d

[Gary Laevsky]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Simon,

On top of that, storage and moving the datasets.  We just set up a 10Gb
fiber optic between acquisition and processing PCs and our central server.
We're estimating 200-250 Tb a year between lightsheet, localization, and
24-72 hour 6D datasets using sCMOS cameras on a disc.

We're also looking at Amazon Glacier (well, more than looking).

I imagine this is easily a workshop.  How much interest is there?

We've been struggling looking for a solution for awhile.

Eagerly following this thread ...

Gary



On Tue, Jul 26, 2016 at 11:26 AM, Watkins, Simon C <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Folks, Data sets are getting bigger and bigger... many hundreds of
> gigabytes commonly.  This is particularly pressing with light sheet data or
> 3D imaging of cleared tissue samples.
> My question for the forum is how are you dealing with this? Many of the
> common packages we all use support CUDA based rendering and processing but
> do any allow multi-CPU processing or a distributed architecture.
> How about hardware solutions?
> I am not so much interested in how data is being moved with this post,
> that really another question and I think more readily solved.  I would like
> to know specifically about quantifying and rendering massive multicolor, 3D
> data sets
> If commercial suppliers wish to contact me offline my direct mail address
> is below.
>
> Simon Watkins Ph.D
> Distinguished Professor and Vice Chair Cell Biology
> Professor Immunology
> Director Center for Biologic Imaging
> University of Pittsburgh
> Bsts 225 3550 terrace st
> Pittsburgh PA 15261
> [hidden email]<mailto:[hidden email]>
> Www.cbi.pitt.edu<http://Www.cbi.pitt.edu>
> 412-352-2277
>

--
Best,

Gary Laevsky, Ph.D.
Director, Confocal Imaging Facility
Nikon Center of Excellence
Dept. of Molecular Biology
Washington Rd.
Princeton University
Princeton, New Jersey, 08544-1014
(O) 609 258 5432
(C) 508 507 1310

[George McNamara]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Tim,

Expansion Microscopy: Yes. Ed Boyden, and see also Chozinski ... Wong,
Vaughan 2016 Nat Meth

http://www.nature.com/nmeth/journal/v13/n6/full/nmeth.3833.html

(and kudos to Nature Methods for going open access!)

4x expansion is 64x larger volume to deal with (big data: deal with it)
AND much more uniform refractive index throughout, so less scattering.
Sure, the protocols may extract all the lipids (most membranes are ~50%
protein, so still a lot of stuff after extraction), most researchers are
interested in any or all of the DNA, RNA and proteins (and
glycoproteins) and post-translational modifications of the proteins.

Recent Nikon research microscope brochure:

CFI Plan Apochromat 40x 0.95NA, 0.14 mm WD (and coverglass correction
collar)
CFI Plan Apochromat 40x 1.00NA, 0.16 mm WD (oil)

CFI Plan Fluor 40x 0.75NA, 0.72 mm WD
CFI Plan Fluor 40x 1.30NA, 0.20 mm WD

So, for these plan apo's, the oil immesion lens has slightly (200 um)
working distance: good, use it (R.I. match the mounting media,
transparent, hopefully low viscosity).

For the Plan Fluor's the dry lens has almost 4x working distance, so on
expansion about the same final specimen thickness.

However, the payoffs for Expansion include:

* better refractive index matching ... less scatter, so more photons end
up in the correct place (the sCMOS detectors). This will also improve
quantitative deconvolution (SVI Huygens or Microvolution GPU deconvolution).

* referring back to original specimen size, 4x Expansion is effectively
4x (in each dimension) better spatial resolution. An NA=3.0 or NA=3.8
lens is not practical, but Expansion results in effective NA = (4 *
0.75NA) for the plan fluor and eNA = (0.95 * 4) for the plan apo.

I also suggest that Expansion will play well with Light Sheet
microscopy, with only the fluorophores in the sheet being susceptible to
photobleaching. Plus, the beam waist of the light sheet can be tweaked
(lower NA) to make it more uniform across a larger specimen distance.

I also note that Adam Hoppe recently published an improved "joint"
spatial deconvolution and spectral unmixing paper:

Three-Dimensional Reconstruction of Three-Way FRET Microscopy Improves
Imaging of Multiple Protein-Protein Interactions.
Scott BL, Hoppe AD.
PLoS One. 2016 Mar 29;11(3):e0152401. doi: 10.1371/journal.pone.0152401.
eCollection 2016.
PMID: 27023704

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152401

Fluorescence resonance energy transfer (FRET) microscopy is a powerful
tool for imaging the interactions between fluorescently tagged proteins
in two-dimensions. For FRET microscopy to reach its full potential, it
must be able to image more than one pair of interacting molecules and
image degradation from out-of-focus light must be reduced. Here we
extend our previous work on the application of maximum likelihood
methods to the 3-dimensional reconstruction of 3-way FRET interactions
within cells. We validated the new method (3D-3Way FRET) by simulation
and fluorescent protein test constructs expressed in cells. In addition,
we improved the computational methods to create a 2-log reduction in
computation time over our previous method (3DFSR). We applied 3D-3Way
FRET to image the 3D subcellular distributions of HIV Gag assembly. Gag
fused to three different FPs (CFP, YFP, and RFP), assembled into
viral-like particles and created punctate FRET signals that become
visible on the cell surface when 3D-3Way FRET was applied to the data.
Control experiments in which YFP-Gag, RFP-Gag and free CFP were
expressed, demonstrated localized FRET between YFP and RFP at sites of
viral assembly that were not associated with CFP. 3D-3Way FRET provides
the first approach for quantifying multiple FRET interactions while
improving the 3D resolution of FRET microscopy data without introducing
bias into the reconstructed estimates. This method should allow
improvement of widefield, confocal and superresolution FRET microscopy data.

//

the Scott & Hoppe 2016 paper was focused on live cells (something to do
before fixation and Expansion), and clearly a "2-log reduction in
computation time over our previous method" is a good thing (that is 2
log10 ~100 fold ... compared to their 2008 slow programming in Matlab
and old PC hardware). OK, so Expansion by 4x (4^3 = 64) brings in a
whole lot more voxels -- and multiplex is even more. So what? We now
live an an era where a single 10 Teraflop GPU is around $1000 (GTX 1080
Founder Edition is under $1000, TITAN X is $1200), so 1 Petaflop is 100
of these cards, about $100,000 (I'm ignoring the server racks, wiring,
electricity and cooling bill: maybe the GPU cluster could be used to
heat the sauna adjacent to your office & gym). Also, the signal is
mostly "present or absent" per voxel at the 3x to 4x (per axis) of most
of the published Expansion Microscopy papers. I head Ed Boyden speak in
Spring 2015 about 20x (per axis) Expansion. I suggest 10x (per axis)
Expansion microscopy with a 20x 0.7 (0.75, 0.8) NA objective lens would
be a nice sweet spot: 1/4 the field of view of 100x lens, "effective" NA
= (0.7 * 10). There are plenty of examples where a given signal is + or
- per pixel (binary: either present or absent), and signal variation per
object really doesn't matter (because due to stochastic binding of the
fluorescent probes), for example, Transferrin Receptor mRNA molecules in
this figure,

http://stellarisgallery.biosearchtech.com/ShipReady-Probe-Sets/i-NtP8Nzs/A

can be counted by eye (conventional fluorescence microscopy) for each
cell if you want to (I have not). That image was done without
quantitative deconvolution, but deconvolution would help in dense mRNA
regions, and Expansion + deconvolution would help even more.


Enjoy,

George


On 7/26/2016 12:58 PM, Feinstein, Timothy N wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Has anyone used the workflow from Amat...Keller (2015) in Nature
> Protocols?  They report (Fig 5) that their new compression/analysis
> toolkit works well with multiple processor cores, especially compared with
> other compression alternatives like LZW or JPEG2000.  That may be a good
> way to make the most of distributed computing resources when those are
> available.
>
> Expansion microscopy (effectively) increases resolution but cuts working
> distance.  Does it help to have larger structures if you need to look
> deeper to see them?
>
> Best,
>
>
> Tim
>
> Timothy Feinstein, Ph.D.
> Research Scientist
> University of Pittsburgh Department of Developmental Biology
>
>
>
>
>
> On 7/26/16, 12:45 PM, "Confocal Microscopy List on behalf of George
> McNamara" <[hidden email] on behalf of
> [hidden email]> wrote:
>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.
>> edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7ctnf8%40PITT.ED
>> U%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c
>> 1&sdata=zCFGxcxeouEzk9bGXxtIR3FZ1aUPhBF8q5JmT9rrif0%3d
>> Post images on
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.
>> com&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9e
>> f9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=6olqkfmArvSQrGctzPKRG0WQbssJUEK8Z
>> ZLdukDXTzA%3d and include the link in your posting.
>> *****
>>
>> Hi Simon,
>>
>> Processing for deconvolution: both Microvolution.com and SVI Huygens
>> support NVidia GPU (microvolution can support more than one, haven't
>> checked for Huygens). Latest NVidia GeForce GTX 1070 and 1080 cards
>> require a power cable from the power supply. NVidia just announced new
>> Titan X (hopefully no power cable needed).
>>
>> For digital slide imaging (IHC, few and more color immunofluorescence) I
>> am starting to use Definiens and have high hopes for Visiopharm.
>>
>> AMIRA 3D software is now part of FEI (which I now think of a the cryo-EM
>> company) which is being acquired by ThermoFisher. I doubt this means
>> that every Floid, Evos, or Cellomics will ship next year with AMIRA (if
>> they did, the big four microscope companies and traditional image
>> analysis companies might then take a hit).
>>
>> I still do not understand CellProfiler. Fiji ImageJ2 is not doing much
>> for me. MetaMorph is showing its age and failed to move into 64-bit
>> scale data (limit of 32000x32000 pixel image size, 65000 objects in
>> Integrated Morphometry Analysis).
>>
>> Multiplex immunofluorescence and/or FISH: Garry Nolan and colleagues
>> should be close (this year) to publishing MIBI-TOF: 40plex, 20x20x~20 nm
>> voxel size, 1 mm^2 in 30 minutes (Garry mentions in his talks that one
>> mAb + "CyTOF" detction polymer:heavy metal is about this volume, so one
>> voxel = one molecule). Think imaging CyTOF. On the fluorescence side, I
>> am looking forward to their CODIS (something like George Church FISSEQ,
>> Xiaowei Zhuang, Cai Long have published similar ... and maybe CODIS and
>> Expansion Microscopy will work well together). ... I can also see modest
>> plex fluorescence and then comprehensive single cell RNAseq, something
>> along the lines of Tirosh 2016 (PubMed 27124452). Some of my thoughts
>> about fluorescence multiplexing to 20plex and beyond (I've not replaced
>> eight with ten since hearing about BD FACsymphony with decagons):
>>
>> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.linke
>> din.com%2fpulse%2fbd-biosciences-listed-tandems-horizon-brilliant-violets-
>> mcnamara&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6
>> %7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=wYGfzZEok0RA5eI3MNyLA2P%2bC4
>> mJwog2xCHuxpWISOY%3d
>>
>>
>> big data analysis:
>>
>> TIBCO Spotfire
>>
>> looks promising.
>>
>> lighter weight and/or harder to use: RapidMiner, Origin Pro (9 million
>> cells vs max 1 million rows for Excel), Tableau for visualization and
>> Tableau for Student(s) is free for one year - just keep finding a new
>> student every year per workstation.
>>
>> For my consulting (between sending job applications) I now use a
>> Yamasaki 40" UHD 3840 x 2160 pixel monitor ($500) for digital slide
>> image analysis. Internet connectivity is pretty good (comcast at home),
>> remote desktop connection to the remote workstation.
>>
>> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.amazo
>> n.com%2fYAMAKASI-O40USUT-Inch-Monitor-10Bit%2fdp%2fB017VWX3U8&data=01%7c01
>> %7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87
>> cc3a526112fd0d%7c1&sdata=%2fBPJHQdoQQN6V6kckROHCV2l4xBshMEp4kD2znPP9p8%3d
>> YAMAKASI O40USUT 40 Inch UHD / 4K Monitor (3840 x 2160) HDMI 2.0, UHD
>> (60Hz), HDCP 2.2, MHL, 10Bit
>> I discovered online how to switch the display from Korean to English.
>> HDMi2.0->DisplayPort cable arrives today since standard HDMI1.x cable
>> limits the monitor to 30Hz. I also found online that HDMI #4 is the only
>> port on the monitor with 60 Hz capability (if HDMI2.0 cable and assuming
>> graphics card works). 40" is a bit too big to easily use two on one
>> system.
>>
>> //
>>
>> In addition to the montly (weekly?) new papers on Light Sheet, I am
>> excited about the Expansion Microscopy. See Ed Boyden's papers 88 and 89
>> downloadable at
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fsyntheticn
>> eurobiology.org%2fpublications&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507
>> a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=RweqB9
>> SKyLpFwTGyjkokHEkOIjXL6Ih79VVZJRK%2f0Eo%3d
>>
>> plus opportunities to combine ExM with light sheet.
>>
>> George
>>
>>
>> On 7/26/2016 10:26 AM, Watkins, Simon C wrote:
>>> *****
>>> To join, leave or search the confocal microscopy listserv, go to:
>>>
>>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn
>>> .edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7ctnf8%40PITT.
>>> EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d
>>> %7c1&sdata=zCFGxcxeouEzk9bGXxtIR3FZ1aUPhBF8q5JmT9rrif0%3d
>>> Post images on
>>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur
>>> .com&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c
>>> 9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=6olqkfmArvSQrGctzPKRG0WQbssJUE
>>> K8ZZLdukDXTzA%3d and include the link in your posting.
>>> *****
>>>
>>> Folks, Data sets are getting bigger and bigger... many hundreds of
>>> gigabytes commonly.  This is particularly pressing with light sheet data
>>> or 3D imaging of cleared tissue samples.
>>> My question for the forum is how are you dealing with this? Many of the
>>> common packages we all use support CUDA based rendering and processing
>>> but do any allow multi-CPU processing or a distributed architecture.
>>> How about hardware solutions?
>>> I am not so much interested in how data is being moved with this post,
>>> that really another question and I think more readily solved.  I would
>>> like to know specifically about quantifying and rendering massive
>>> multicolor, 3D data sets
>>> If commercial suppliers wish to contact me offline my direct mail
>>> address is below.
>>>
>>> Simon Watkins Ph.D
>>> Distinguished Professor and Vice Chair Cell Biology
>>> Professor Immunology
>>> Director Center for Biologic Imaging
>>> University of Pittsburgh
>>> Bsts 225 3550 terrace st
>>> Pittsburgh PA 15261
>>> [hidden email]<mailto:[hidden email]>
>>>
>>> https://na01.safelinks.protection.outlook.com/?url=Www.cbi.pitt.edu&data=
>>> 01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b5745fb6%7c9ef9f489e0
>>> a04eeb87cc3a526112fd0d%7c1&sdata=EwPWpth3tusWtzmY5Er6JXMPi2c1IL9QvYM9hqQI
>>> rfM%3d<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fWw
>>> w.cbi.pitt.edu&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a94cfb353a08d3b
>>> 5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=ejZynFRdwaWxJIESmghC
>>> SUX1iDF4tqXYqrtU4XQ2o1s%3d>
>>> 412-352-2277
>> --
>>
>>
>> George McNamara, PhD
>> Houston, TX 77054
>> [hidden email]
>> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.linke
>> din.com%2fin%2fgeorgemcnamara&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a
>> 94cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=9TyvMQA
>> YK7XWLKn3vHttEg1q%2bN%2fAHYgEG9T2xiN1lYc%3d
>> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fworks.bep
>> ress.com%2fgmcnamara%2f75%2f&data=01%7c01%7ctnf8%40PITT.EDU%7cd327c76507a9
>> 4cfb353a08d3b5745fb6%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=Ruc%2fyM
>> 5MvOXiUpqTtIXXULVMyHSyb4U5Y2DsR278MhA%3d

--


George McNamara, PhD
Houston, TX 77054
[hidden email]
https://www.linkedin.com/in/georgemcnamara
https://works.bepress.com/gmcnamara/75/

[Kurt Thorn]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

On 7/26/2016 10:58 AM, Feinstein, Timothy N wrote:
>
> Expansion microscopy (effectively) increases resolution but cuts working
> distance.  Does it help to have larger structures if you need to look
> deeper to see them?
>
>
We have imaged a few expanded samples. One thing I did not appreciate
until we started imaging them is that the refractive index of these
samples is really close to water, so water immersion objectives work
very well with them. We were able to image through 250 um of expanded
tissue using a Nikon 40x / 1.15 WI lens on a CSU-W1 confocal. We could
have gone thicker, but that was all the sample we had.  The images 250
um in looked nearly as good as the ones at the surface of the section.

Obviously your resolution is a bit lower than a 1.4 NA oil lens, but
that might be the price you have to pay.

Kurt


--
Kurt Thorn
Associate Professor
Director, Nikon Imaging Center
http://thornlab.ucsf.edu/
http://nic.ucsf.edu/blog/

[Paul Paroutis]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

We have been facing the same issue since the purchase of our Zeiss Lightsheet system. On the commercial side of things, Arivis has worked well for us and I would recommend giving that a shot. On the deconvolution side, we recently purchased the Huygens deconvolution module and it has given us nice results. We had also tested the Microvolution software and were really impressed at the speed and quality of deconvolution - the price tag put it out of our range for the time being, but it's definitely worth exploring.

[Douglas Richardson]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I'll echo Paul's endorsement of Arivis for 3D data sets and George's
suggestion regarding Visiopharm for 2D data sets (I really love that it
doesn't duplicate the data into yet another proprietary file type).


However, theses are both expensive and there are open source options as
well.  One of our groups has a great open-source work flow for imaging and
registering cleared brains (imaged & registered >80 cleared brains, ~150TB
of data). Here is the reference:
http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-posterior-striatum-form-ananatomically-distinct.
The Tessier-Lavigne lab just released a computational method (ClearMap
http://www.sciencedirect.com/science/article/pii/S0092867416305554 for a
similar process, as has the Ueda group with their CUBIC method (
http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
although these both mainly deal with ultra-microscope data which isn't as
intensive as other forms of lightsheet.

Big data viewer in Fiji and Vaa3D are also good open source options for
viewing the data.

On the data storage side, the above mentioned publication was done mainly
with a filing cabinet full of 2TB USB 3.0 external hard drives.  Since
then, we've run 10Gbit optical fiber to all of our microscopes and
workstations.  Most importantly, this 10Gbit connection goes right through
to our expandable storage server downtown.

I think the two big lessons we've learned are the following:

1) Make sure your storage is expandable, you'll never have enough. We're
currently at 250TB in a LUSTER configuration with plans to push into PTs
soon.
2) You will always need to move data, make sure your connections are fast.
We have a 3 tier system: 1) Microscope acquisition computer > 2) Processing
workstations > 3) Long-term storage server.  Connections to the cloud are
not fast enough, so I don't feel this is an option.

Finally, many versions of commercial microscope acquisition software are
unable to directly save data to network storage (or external drives) no
matter how fast the connection. This is a feature we need to push the
manufacturers for or else you'll always be limited to the storage space on
your acquisition computer.

-Doug

On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> We have been facing the same issue since the purchase of our Zeiss
> Lightsheet system. On the commercial side of things, Arivis has worked well
> for us and I would recommend giving that a shot. On the deconvolution side,
> we recently purchased the Huygens deconvolution module and it has given us
> nice results. We had also tested the Microvolution software and were really
> impressed at the speed and quality of deconvolution - the price tag put it
> out of our range for the time being, but it's definitely worth exploring.
>

[0000001ed7f52e4a-dmarc-request]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Would it not be much better to perform the data analysis on a scalable cluster which has fast connection to the storage instead of moving data around? We need to push software companies to make their solutions run on these machines. Instead of buying ever bigger analysis workstations which are obsolete after a few years, one would just buy computing time. The cluster can be shared with bioinformatics groups.

My take on storage is that you need to have a cheap archive, otherwise there will be a point at which you run out of money to keep the ever expanding storage.

Best wishes

Andreas

-----Original Message-----
From: "Douglas Richardson" <[hidden email]>
Sent: ‎27/‎07/‎2016 15:34
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I'll echo Paul's endorsement of Arivis for 3D data sets and George's
suggestion regarding Visiopharm for 2D data sets (I really love that it
doesn't duplicate the data into yet another proprietary file type).


However, theses are both expensive and there are open source options as
well.  One of our groups has a great open-source work flow for imaging and
registering cleared brains (imaged & registered >80 cleared brains, ~150TB
of data). Here is the reference:
http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-posterior-striatum-form-ananatomically-distinct.
The Tessier-Lavigne lab just released a computational method (ClearMap
http://www.sciencedirect.com/science/article/pii/S0092867416305554 for a
similar process, as has the Ueda group with their CUBIC method (
http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
although these both mainly deal with ultra-microscope data which isn't as
intensive as other forms of lightsheet.

Big data viewer in Fiji and Vaa3D are also good open source options for
viewing the data.

On the data storage side, the above mentioned publication was done mainly
with a filing cabinet full of 2TB USB 3.0 external hard drives.  Since
then, we've run 10Gbit optical fiber to all of our microscopes and
workstations.  Most importantly, this 10Gbit connection goes right through
to our expandable storage server downtown.

I think the two big lessons we've learned are the following:

1) Make sure your storage is expandable, you'll never have enough. We're
currently at 250TB in a LUSTER configuration with plans to push into PTs
soon.
2) You will always need to move data, make sure your connections are fast.
We have a 3 tier system: 1) Microscope acquisition computer > 2) Processing
workstations > 3) Long-term storage server.  Connections to the cloud are
not fast enough, so I don't feel this is an option.

Finally, many versions of commercial microscope acquisition software are
unable to directly save data to network storage (or external drives) no
matter how fast the connection. This is a feature we need to push the
manufacturers for or else you'll always be limited to the storage space on
your acquisition computer.

-Doug

On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> We have been facing the same issue since the purchase of our Zeiss
> Lightsheet system. On the commercial side of things, Arivis has worked well
> for us and I would recommend giving that a shot. On the deconvolution side,
> we recently purchased the Huygens deconvolution module and it has given us
> nice results. We had also tested the Microvolution software and were really
> impressed at the speed and quality of deconvolution - the price tag put it
> out of our range for the time being, but it's definitely worth exploring.
>

[G. Esteban Fernandez]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I just wanted to echo the praises for Arivis and MicroVolution.

My favorite for 3D work is Imaris but it can't handle big data; when
possible I downsample large datasets and work in Imaris. Amira can handle
big data but in my experience it crashed more often than Arivis plus I
prefer the user interface in Arivis.

MicroVolution results were comparable to Huygens and AutoQuant in my hands
(qualitatively, I didn't do rigorous quantitative comparisons) in about
1/60 of the time with a lower end GPU. I mostly looked at confocal
point-scanning data and didn't try truly big data. MicroVolution is limited
to datasets <RAM, so you subvolume yourself before deconvolving.

-Esteban

On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" <
[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Would it not be much better to perform the data analysis on a scalable
> cluster which has fast connection to the storage instead of moving data
> around? We need to push software companies to make their solutions run on
> these machines. Instead of buying ever bigger analysis workstations which
> are obsolete after a few years, one would just buy computing time. The
> cluster can be shared with bioinformatics groups.
>
> My take on storage is that you need to have a cheap archive, otherwise
> there will be a point at which you run out of money to keep the ever
> expanding storage.
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "Douglas Richardson" <[hidden email]>
> Sent: ‎27/‎07/‎2016 15:34
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
> suggestion regarding Visiopharm for 2D data sets (I really love that it
> doesn't duplicate the data into yet another proprietary file type).
>
>
> However, theses are both expensive and there are open source options as
> well.  One of our groups has a great open-source work flow for imaging and
> registering cleared brains (imaged & registered >80 cleared brains, ~150TB
> of data). Here is the reference:
>
> http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-posterior-striatum-form-ananatomically-distinct
> .
> The Tessier-Lavigne lab just released a computational method (ClearMap
> http://www.sciencedirect.com/science/article/pii/S0092867416305554 for a
> similar process, as has the Ueda group with their CUBIC method (
> http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
> although these both mainly deal with ultra-microscope data which isn't as
> intensive as other forms of lightsheet.
>
> Big data viewer in Fiji and Vaa3D are also good open source options for
> viewing the data.
>
> On the data storage side, the above mentioned publication was done mainly
> with a filing cabinet full of 2TB USB 3.0 external hard drives.  Since
> then, we've run 10Gbit optical fiber to all of our microscopes and
> workstations.  Most importantly, this 10Gbit connection goes right through
> to our expandable storage server downtown.
>
> I think the two big lessons we've learned are the following:
>
> 1) Make sure your storage is expandable, you'll never have enough. We're
> currently at 250TB in a LUSTER configuration with plans to push into PTs
> soon.
> 2) You will always need to move data, make sure your connections are fast.
> We have a 3 tier system: 1) Microscope acquisition computer > 2) Processing
> workstations > 3) Long-term storage server.  Connections to the cloud are
> not fast enough, so I don't feel this is an option.
>
> Finally, many versions of commercial microscope acquisition software are
> unable to directly save data to network storage (or external drives) no
> matter how fast the connection. This is a feature we need to push the
> manufacturers for or else you'll always be limited to the storage space on
> your acquisition computer.
>
> -Doug
>
> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis <[hidden email]>
> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> > Post images on http://www.imgur.com and include the link in your
> posting.
> > *****
> >
> > We have been facing the same issue since the purchase of our Zeiss
> > Lightsheet system. On the commercial side of things, Arivis has worked
> well
> > for us and I would recommend giving that a shot. On the deconvolution
> side,
> > we recently purchased the Huygens deconvolution module and it has given
> us
> > nice results. We had also tested the Microvolution software and were
> really
> > impressed at the speed and quality of deconvolution - the price tag put
> it
> > out of our range for the time being, but it's definitely worth exploring.
> >
>

[DENNIS Andrew]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Esteban,

I work at Andor/Bitpane so you may consider this to be a commercial response..

I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?

Andrew


________________________________________
From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
Sent: 29 July 2016 20:43
To: [hidden email]
Subject: Re: analyzing really big data sets

EXTERNAL EMAIL

ATTACHMENT ADVISORY

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I just wanted to echo the praises for Arivis and MicroVolution.

My favorite for 3D work is Imaris but it can't handle big data; when
possible I downsample large datasets and work in Imaris. Amira can handle
big data but in my experience it crashed more often than Arivis plus I
prefer the user interface in Arivis.

MicroVolution results were comparable to Huygens and AutoQuant in my hands
(qualitatively, I didn't do rigorous quantitative comparisons) in about
1/60 of the time with a lower end GPU. I mostly looked at confocal
point-scanning data and didn't try truly big data. MicroVolution is limited
to datasets <RAM, so you subvolume yourself before deconvolving.

-Esteban

On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" <
[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Would it not be much better to perform the data analysis on a scalable
> cluster which has fast connection to the storage instead of moving data
> around? We need to push software companies to make their solutions run on
> these machines. Instead of buying ever bigger analysis workstations which
> are obsolete after a few years, one would just buy computing time. The
> cluster can be shared with bioinformatics groups.
>
> My take on storage is that you need to have a cheap archive, otherwise
> there will be a point at which you run out of money to keep the ever
> expanding storage.
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "Douglas Richardson" <[hidden email]>
> Sent: ‎27/‎07/‎2016 15:34
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
> suggestion regarding Visiopharm for 2D data sets (I really love that it
> doesn't duplicate the data into yet another proprietary file type).
>
>
> However, theses are both expensive and there are open source options as
> well.  One of our groups has a great open-source work flow for imaging and
> registering cleared brains (imaged & registered >80 cleared brains, ~150TB
> of data). Here is the reference:
>
> http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-posterior-striatum-form-ananatomically-distinct
> .
> The Tessier-Lavigne lab just released a computational method (ClearMap
> http://www.sciencedirect.com/science/article/pii/S0092867416305554 for a
> similar process, as has the Ueda group with their CUBIC method (
> http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
> although these both mainly deal with ultra-microscope data which isn't as
> intensive as other forms of lightsheet.
>
> Big data viewer in Fiji and Vaa3D are also good open source options for
> viewing the data.
>
> On the data storage side, the above mentioned publication was done mainly
> with a filing cabinet full of 2TB USB 3.0 external hard drives.  Since
> then, we've run 10Gbit optical fiber to all of our microscopes and
> workstations.  Most importantly, this 10Gbit connection goes right through
> to our expandable storage server downtown.
>
> I think the two big lessons we've learned are the following:
>
> 1) Make sure your storage is expandable, you'll never have enough. We're
> currently at 250TB in a LUSTER configuration with plans to push into PTs
> soon.
> 2) You will always need to move data, make sure your connections are fast.
> We have a 3 tier system: 1) Microscope acquisition computer > 2) Processing
> workstations > 3) Long-term storage server.  Connections to the cloud are
> not fast enough, so I don't feel this is an option.
>
> Finally, many versions of commercial microscope acquisition software are
> unable to directly save data to network storage (or external drives) no
> matter how fast the connection. This is a feature we need to push the
> manufacturers for or else you'll always be limited to the storage space on
> your acquisition computer.
>
> -Doug
>
> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis <[hidden email]>
> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> > Post images on http://www.imgur.com and include the link in your
> posting.
> > *****
> >
> > We have been facing the same issue since the purchase of our Zeiss
> > Lightsheet system. On the commercial side of things, Arivis has worked
> well
> > for us and I would recommend giving that a shot. On the deconvolution
> side,
> > we recently purchased the Huygens deconvolution module and it has given
> us
> > nice results. We had also tested the Microvolution software and were
> really
> > impressed at the speed and quality of deconvolution - the price tag put
> it
> > out of our range for the time being, but it's definitely worth exploring.
> >
>

+++Scanned for Viruses by ForcePoint+++


___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to www.oxinst.com/email-statement for regulatory information.

[DENNIS Andrew]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Sorry Typo in an embarrassing part of my last message,

It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "

-----Original Message-----
From: DENNIS Andrew
Sent: 29 July 2016 23:17
To: Confocal Microscopy List <[hidden email]>
Subject: RE: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Esteban,

I work at Andor/Bitpane so you may consider this to be a commercial response..

I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?

Andrew


________________________________________
From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
Sent: 29 July 2016 20:43
To: [hidden email]
Subject: Re: analyzing really big data sets

EXTERNAL EMAIL

ATTACHMENT ADVISORY

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I just wanted to echo the praises for Arivis and MicroVolution.

My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.

MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.

-Esteban

On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Would it not be much better to perform the data analysis on a scalable
> cluster which has fast connection to the storage instead of moving
> data around? We need to push software companies to make their
> solutions run on these machines. Instead of buying ever bigger
> analysis workstations which are obsolete after a few years, one would
> just buy computing time. The cluster can be shared with bioinformatics groups.
>
> My take on storage is that you need to have a cheap archive, otherwise
> there will be a point at which you run out of money to keep the ever
> expanding storage.
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "Douglas Richardson" <[hidden email]>
> Sent: ‎27/‎07/‎2016 15:34
> To: "[hidden email]"
> <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
> suggestion regarding Visiopharm for 2D data sets (I really love that
> it doesn't duplicate the data into yet another proprietary file type).
>
>
> However, theses are both expensive and there are open source options
> as well.  One of our groups has a great open-source work flow for
> imaging and registering cleared brains (imaged & registered >80
> cleared brains, ~150TB of data). Here is the reference:
>
> http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-p
> osterior-striatum-form-ananatomically-distinct
> .
> The Tessier-Lavigne lab just released a computational method (ClearMap
> http://www.sciencedirect.com/science/article/pii/S0092867416305554 for
> a similar process, as has the Ueda group with their CUBIC method (
> http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
> although these both mainly deal with ultra-microscope data which isn't
> as intensive as other forms of lightsheet.
>
> Big data viewer in Fiji and Vaa3D are also good open source options
> for viewing the data.
>
> On the data storage side, the above mentioned publication was done
> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.  
> Since then, we've run 10Gbit optical fiber to all of our microscopes
> and workstations.  Most importantly, this 10Gbit connection goes right
> through to our expandable storage server downtown.
>
> I think the two big lessons we've learned are the following:
>
> 1) Make sure your storage is expandable, you'll never have enough.
> We're currently at 250TB in a LUSTER configuration with plans to push
> into PTs soon.
> 2) You will always need to move data, make sure your connections are fast.
> We have a 3 tier system: 1) Microscope acquisition computer > 2)
> Processing workstations > 3) Long-term storage server.  Connections to
> the cloud are not fast enough, so I don't feel this is an option.
>
> Finally, many versions of commercial microscope acquisition software
> are unable to directly save data to network storage (or external
> drives) no matter how fast the connection. This is a feature we need
> to push the manufacturers for or else you'll always be limited to the
> storage space on your acquisition computer.
>
> -Doug
>
> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
> <[hidden email]>
> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> > Post images on http://www.imgur.com and include the link in your
> posting.
> > *****
> >
> > We have been facing the same issue since the purchase of our Zeiss
> > Lightsheet system. On the commercial side of things, Arivis has
> > worked
> well
> > for us and I would recommend giving that a shot. On the
> > deconvolution
> side,
> > we recently purchased the Huygens deconvolution module and it has
> > given
> us
> > nice results. We had also tested the Microvolution software and were
> really
> > impressed at the speed and quality of deconvolution - the price tag
> > put
> it
> > out of our range for the time being, but it's definitely worth exploring.
> >
>

+++Scanned for Viruses by ForcePoint+++


___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to www.oxinst.com/email-statement for regulatory information.

[0000001ed7f52e4a-dmarc-request]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Andrew,

Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?

Best wishes

Andreas

-----Original Message-----
From: "DENNIS Andrew" <[hidden email]>
Sent: ‎29/‎07/‎2016 23:39
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Sorry Typo in an embarrassing part of my last message,

It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "

-----Original Message-----
From: DENNIS Andrew
Sent: 29 July 2016 23:17
To: Confocal Microscopy List <[hidden email]>
Subject: RE: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Esteban,

I work at Andor/Bitpane so you may consider this to be a commercial response..

I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?

Andrew


________________________________________
From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
Sent: 29 July 2016 20:43
To: [hidden email]
Subject: Re: analyzing really big data sets

EXTERNAL EMAIL

ATTACHMENT ADVISORY

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I just wanted to echo the praises for Arivis and MicroVolution.

My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.

MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.

-Esteban

On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Would it not be much better to perform the data analysis on a scalable
> cluster which has fast connection to the storage instead of moving
> data around? We need to push software companies to make their
> solutions run on these machines. Instead of buying ever bigger
> analysis workstations which are obsolete after a few years, one would
> just buy computing time. The cluster can be shared with bioinformatics groups.
>
> My take on storage is that you need to have a cheap archive, otherwise
> there will be a point at which you run out of money to keep the ever
> expanding storage.
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "Douglas Richardson" <[hidden email]>
> Sent: ‎27/‎07/‎2016 15:34
> To: "[hidden email]"
> <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
> suggestion regarding Visiopharm for 2D data sets (I really love that
> it doesn't duplicate the data into yet another proprietary file type).
>
>
> However, theses are both expensive and there are open source options
> as well.  One of our groups has a great open-source work flow for
> imaging and registering cleared brains (imaged & registered >80
> cleared brains, ~150TB of data). Here is the reference:
>
> http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-p
> osterior-striatum-form-ananatomically-distinct
> .
> The Tessier-Lavigne lab just released a computational method (ClearMap
> http://www.sciencedirect.com/science/article/pii/S0092867416305554 for
> a similar process, as has the Ueda group with their CUBIC method (
> http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
> although these both mainly deal with ultra-microscope data which isn't
> as intensive as other forms of lightsheet.
>
> Big data viewer in Fiji and Vaa3D are also good open source options
> for viewing the data.
>
> On the data storage side, the above mentioned publication was done
> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.  
> Since then, we've run 10Gbit optical fiber to all of our microscopes
> and workstations.  Most importantly, this 10Gbit connection goes right
> through to our expandable storage server downtown.
>
> I think the two big lessons we've learned are the following:
>
> 1) Make sure your storage is expandable, you'll never have enough.
> We're currently at 250TB in a LUSTER configuration with plans to push
> into PTs soon.
> 2) You will always need to move data, make sure your connections are fast.
> We have a 3 tier system: 1) Microscope acquisition computer > 2)
> Processing workstations > 3) Long-term storage server.  Connections to
> the cloud are not fast enough, so I don't feel this is an option.
>
> Finally, many versions of commercial microscope acquisition software
> are unable to directly save data to network storage (or external
> drives) no matter how fast the connection. This is a feature we need
> to push the manufacturers for or else you'll always be limited to the
> storage space on your acquisition computer.
>
> -Doug
>
> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
> <[hidden email]>
> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> > Post images on http://www.imgur.com and include the link in your
> posting.
> > *****
> >
> > We have been facing the same issue since the purchase of our Zeiss
> > Lightsheet system. On the commercial side of things, Arivis has
> > worked
> well
> > for us and I would recommend giving that a shot. On the
> > deconvolution
> side,
> > we recently purchased the Huygens deconvolution module and it has
> > given
> us
> > nice results. We had also tested the Microvolution software and were
> really
> > impressed at the speed and quality of deconvolution - the price tag
> > put
> it
> > out of our range for the time being, but it's definitely worth exploring.
> >
>

+++Scanned for Viruses by ForcePoint+++


___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to www.oxinst.com/email-statement for regulatory information.

[Watkins, Simon C-2]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I am glad this question has raised such interest. Importantly though there are really two questions here.  Rendering and displaying data so that it can be visually plumbed and quantitation of structure and events within the rendered volume using potentially complex combinations of filters and segmentation tools.  For my group this is where problems really start

Sent from my iPhone

> On Jul 30, 2016, at 8:41 AM, Andreas Bruckbauer <[hidden email]> wrote:
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> Dear Andrew,
>
> Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "DENNIS Andrew" <[hidden email]>
> Sent: ‎29/‎07/‎2016 23:39
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> Sorry Typo in an embarrassing part of my last message,
>
> It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "
>
> -----Original Message-----
> From: DENNIS Andrew
> Sent: 29 July 2016 23:17
> To: Confocal Microscopy List <[hidden email]>
> Subject: RE: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> Hi Esteban,
>
> I work at Andor/Bitpane so you may consider this to be a commercial response..
>
> I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?
>
> Andrew
>
>
> ________________________________________
> From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
> Sent: 29 July 2016 20:43
> To: [hidden email]
> Subject: Re: analyzing really big data sets
>
> EXTERNAL EMAIL
>
> ATTACHMENT ADVISORY
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> I just wanted to echo the praises for Arivis and MicroVolution.
>
> My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.
>
> MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
> 1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.
>
> -Esteban
>
>> On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
>> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
>> *****
>>
>> Would it not be much better to perform the data analysis on a scalable
>> cluster which has fast connection to the storage instead of moving
>> data around? We need to push software companies to make their
>> solutions run on these machines. Instead of buying ever bigger
>> analysis workstations which are obsolete after a few years, one would
>> just buy computing time. The cluster can be shared with bioinformatics groups.
>>
>> My take on storage is that you need to have a cheap archive, otherwise
>> there will be a point at which you run out of money to keep the ever
>> expanding storage.
>>
>> Best wishes
>>
>> Andreas
>>
>> -----Original Message-----
>> From: "Douglas Richardson" <[hidden email]>
>> Sent: ‎27/‎07/‎2016 15:34
>> To: "[hidden email]"
>> <[hidden email]>
>> Subject: Re: analyzing really big data sets
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
>> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
>> *****
>>
>> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
>> suggestion regarding Visiopharm for 2D data sets (I really love that
>> it doesn't duplicate the data into yet another proprietary file type).
>>
>>
>> However, theses are both expensive and there are open source options
>> as well.  One of our groups has a great open-source work flow for
>> imaging and registering cleared brains (imaged & registered >80
>> cleared brains, ~150TB of data). Here is the reference:
>>
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhcbi.fas.harvard.edu%2fpublications%2fdopamine-neurons-projecting-p&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=tlmNC9Erq98MzwNvJze0ir8hWU7gDI3DXqBr90y8zdU%3d
>> osterior-striatum-form-ananatomically-distinct
>> .
>> The Tessier-Lavigne lab just released a computational method (ClearMap
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.sciencedirect.com%2fscience%2farticle%2fpii%2fS0092867416305554&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=EKbzfoCH4%2fenjdyFiwH0XaEFCbTDMITwh7xoq%2bVcyDQ%3d for
>> a similar process, as has the Ueda group with their CUBIC method (
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.nature.com%2fnprot%2fjournal%2fv10%2fn11%2ffull%2fnprot.2015.085.html&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=Jbk4NsFW4QOz2H%2fWFoM83EVCu3ybSUCaesUtCxx58hI%3d),
>> although these both mainly deal with ultra-microscope data which isn't
>> as intensive as other forms of lightsheet.
>>
>> Big data viewer in Fiji and Vaa3D are also good open source options
>> for viewing the data.
>>
>> On the data storage side, the above mentioned publication was done
>> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.  
>> Since then, we've run 10Gbit optical fiber to all of our microscopes
>> and workstations.  Most importantly, this 10Gbit connection goes right
>> through to our expandable storage server downtown.
>>
>> I think the two big lessons we've learned are the following:
>>
>> 1) Make sure your storage is expandable, you'll never have enough.
>> We're currently at 250TB in a LUSTER configuration with plans to push
>> into PTs soon.
>> 2) You will always need to move data, make sure your connections are fast.
>> We have a 3 tier system: 1) Microscope acquisition computer > 2)
>> Processing workstations > 3) Long-term storage server.  Connections to
>> the cloud are not fast enough, so I don't feel this is an option.
>>
>> Finally, many versions of commercial microscope acquisition software
>> are unable to directly save data to network storage (or external
>> drives) no matter how fast the connection. This is a feature we need
>> to push the manufacturers for or else you'll always be limited to the
>> storage space on your acquisition computer.
>>
>> -Doug
>>
>> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
>> <[hidden email]>
>> wrote:
>>
>>> *****
>>> To join, leave or search the confocal microscopy listserv, go to:
>>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
>>> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your
>> posting.
>>> *****
>>>
>>> We have been facing the same issue since the purchase of our Zeiss
>>> Lightsheet system. On the commercial side of things, Arivis has
>>> worked
>> well
>>> for us and I would recommend giving that a shot. On the
>>> deconvolution
>> side,
>>> we recently purchased the Huygens deconvolution module and it has
>>> given
>> us
>>> nice results. We had also tested the Microvolution software and were
>> really
>>> impressed at the speed and quality of deconvolution - the price tag
>>> put
>> it
>>> out of our range for the time being, but it's definitely worth exploring.
>
>
> +++Scanned for Viruses by ForcePoint+++
>
>
> ___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to https://na01.safelinks.protection.outlook.com/?url=www.oxinst.com%2femail-statement&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=re2O%2f41spaRqGYyhvCPIBjfHEfm4dsISa7Exfc04adg%3d for regulatory information.

[0000001ed7f52e4a-dmarc-request]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Another thing which bothers me (currently for data sets of moderate size e.g. 10 Gb) is that I first have to save the complete data set on the acquisition computer, then open it on the analysis computer and when it is fully loaded, start the analysis. It would be less time consuming if I could already start loading and analysing  the first time points while the last ones are still being saved or even not yet measured. This should be possible, but is not implemented in todays microscope systems.

Best wishes

Andreas

-----Original Message-----
From: "Watkins, Simon C" <[hidden email]>
Sent: ‎30/‎07/‎2016 10:38
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I am glad this question has raised such interest. Importantly though there are really two questions here.  Rendering and displaying data so that it can be visually plumbed and quantitation of structure and events within the rendered volume using potentially complex combinations of filters and segmentation tools.  For my group this is where problems really start

Sent from my iPhone

> On Jul 30, 2016, at 8:41 AM, Andreas Bruckbauer <[hidden email]> wrote:
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> Dear Andrew,
>
> Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "DENNIS Andrew" <[hidden email]>
> Sent: ‎29/‎07/‎2016 23:39
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> Sorry Typo in an embarrassing part of my last message,
>
> It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "
>
> -----Original Message-----
> From: DENNIS Andrew
> Sent: 29 July 2016 23:17
> To: Confocal Microscopy List <[hidden email]>
> Subject: RE: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> Hi Esteban,
>
> I work at Andor/Bitpane so you may consider this to be a commercial response..
>
> I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?
>
> Andrew
>
>
> ________________________________________
> From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
> Sent: 29 July 2016 20:43
> To: [hidden email]
> Subject: Re: analyzing really big data sets
>
> EXTERNAL EMAIL
>
> ATTACHMENT ADVISORY
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
> *****
>
> I just wanted to echo the praises for Arivis and MicroVolution.
>
> My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.
>
> MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
> 1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.
>
> -Esteban
>
>> On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
>> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
>> *****
>>
>> Would it not be much better to perform the data analysis on a scalable
>> cluster which has fast connection to the storage instead of moving
>> data around? We need to push software companies to make their
>> solutions run on these machines. Instead of buying ever bigger
>> analysis workstations which are obsolete after a few years, one would
>> just buy computing time. The cluster can be shared with bioinformatics groups.
>>
>> My take on storage is that you need to have a cheap archive, otherwise
>> there will be a point at which you run out of money to keep the ever
>> expanding storage.
>>
>> Best wishes
>>
>> Andreas
>>
>> -----Original Message-----
>> From: "Douglas Richardson" <[hidden email]>
>> Sent: ‎27/‎07/‎2016 15:34
>> To: "[hidden email]"
>> <[hidden email]>
>> Subject: Re: analyzing really big data sets
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
>> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your posting.
>> *****
>>
>> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
>> suggestion regarding Visiopharm for 2D data sets (I really love that
>> it doesn't duplicate the data into yet another proprietary file type).
>>
>>
>> However, theses are both expensive and there are open source options
>> as well.  One of our groups has a great open-source work flow for
>> imaging and registering cleared brains (imaged & registered >80
>> cleared brains, ~150TB of data). Here is the reference:
>>
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhcbi.fas.harvard.edu%2fpublications%2fdopamine-neurons-projecting-p&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=tlmNC9Erq98MzwNvJze0ir8hWU7gDI3DXqBr90y8zdU%3d
>> osterior-striatum-form-ananatomically-distinct
>> .
>> The Tessier-Lavigne lab just released a computational method (ClearMap
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.sciencedirect.com%2fscience%2farticle%2fpii%2fS0092867416305554&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=EKbzfoCH4%2fenjdyFiwH0XaEFCbTDMITwh7xoq%2bVcyDQ%3d for
>> a similar process, as has the Ueda group with their CUBIC method (
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.nature.com%2fnprot%2fjournal%2fv10%2fn11%2ffull%2fnprot.2015.085.html&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=Jbk4NsFW4QOz2H%2fWFoM83EVCu3ybSUCaesUtCxx58hI%3d),
>> although these both mainly deal with ultra-microscope data which isn't
>> as intensive as other forms of lightsheet.
>>
>> Big data viewer in Fiji and Vaa3D are also good open source options
>> for viewing the data.
>>
>> On the data storage side, the above mentioned publication was done
>> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.  
>> Since then, we've run 10Gbit optical fiber to all of our microscopes
>> and workstations.  Most importantly, this 10Gbit connection goes right
>> through to our expandable storage server downtown.
>>
>> I think the two big lessons we've learned are the following:
>>
>> 1) Make sure your storage is expandable, you'll never have enough.
>> We're currently at 250TB in a LUSTER configuration with plans to push
>> into PTs soon.
>> 2) You will always need to move data, make sure your connections are fast.
>> We have a 3 tier system: 1) Microscope acquisition computer > 2)
>> Processing workstations > 3) Long-term storage server.  Connections to
>> the cloud are not fast enough, so I don't feel this is an option.
>>
>> Finally, many versions of commercial microscope acquisition software
>> are unable to directly save data to network storage (or external
>> drives) no matter how fast the connection. This is a feature we need
>> to push the manufacturers for or else you'll always be limited to the
>> storage space on your acquisition computer.
>>
>> -Doug
>>
>> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
>> <[hidden email]>
>> wrote:
>>
>>> *****
>>> To join, leave or search the confocal microscopy listserv, go to:
>>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.umn.edu%2fcgi-bin%2fwa%3fA0%3dconfocalmicroscopy&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=t1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS%2bO3BGya%2buk%3d
>>> Post images on https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.imgur.com&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=GgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44%3d and include the link in your
>> posting.
>>> *****
>>>
>>> We have been facing the same issue since the purchase of our Zeiss
>>> Lightsheet system. On the commercial side of things, Arivis has
>>> worked
>> well
>>> for us and I would recommend giving that a shot. On the
>>> deconvolution
>> side,
>>> we recently purchased the Huygens deconvolution module and it has
>>> given
>> us
>>> nice results. We had also tested the Microvolution software and were
>> really
>>> impressed at the speed and quality of deconvolution - the price tag
>>> put
>> it
>>> out of our range for the time being, but it's definitely worth exploring.
>
>
> +++Scanned for Viruses by ForcePoint+++
>
>
> ___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to https://na01.safelinks.protection.outlook.com/?url=www.oxinst.com%2femail-statement&data=01%7c01%7csimon.watkins%40PITT.EDU%7c40add1dcc1894c2995a908d3b84cec29%7c9ef9f489e0a04eeb87cc3a526112fd0d%7c1&sdata=re2O%2f41spaRqGYyhvCPIBjfHEfm4dsISa7Exfc04adg%3d for regulatory information.

[mcammer]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

When we do an overnight experiment looking at embryos by lightsheet, the datasets are typically in the 800GB to 1.2 TB range.  We deal with this by having each timepoint of each embryo saved as a separate czi file.  Similar for Nikon Zseries multiple color multiple position overnight timelapse; we save each position as a separate nd2 file and between timepoints we can access each position to date.  To solve this problem with the Zeiss systems, we have Zen export as individual tifs during image collection.  Micromanager also will do this.  So as long as you don't mind have to deal with 80,000 files, you can work around the problem of accessing the beginning of a sequence before the end has happened yet.


_________________________________________
Michael Cammer, Optical Microscopy Specialist
http://ocs.med.nyu.edu/microscopy
http://microscopynotes.com/
Cell: [phone]

________________________________________
From: Confocal Microscopy List [[hidden email]] on behalf of Andreas Bruckbauer [[hidden email]]
Sent: Saturday, July 30, 2016 9:46 AM
To: [hidden email]
Subject: Re: analyzing really big data sets


Another thing which bothers me (currently for data sets of moderate size e.g. 10 Gb) is that I first have to save the complete data set on the acquisition computer, then open it on the analysis computer and when it is fully loaded, start the analysis. It would be less time consuming if I could already start loading and analysing  the first time points while the last ones are still being saved or even not yet measured. This should be possible, but is not implemented in todays microscope systems.

Best wishes

Andreas

-----Original Message-----
From: "Watkins, Simon C" <[hidden email]>
Sent: ‎30/‎07/‎2016 10:38
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=gXXzZP8amUxJuuEYayueruKp-7Ulp0d9NTTdM0UVzJc&e=
Post images on https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=963fE12TwWxn_THDdJAbx0e9Pgcr3w9ZfDxgtJAm-_8&e=  and include the link in your posting.
*****

I am glad this question has raised such interest. Importantly though there are really two questions here.  Rendering and displaying data so that it can be visually plumbed and quantitation of structure and events within the rendered volume using potentially complex combinations of filters and segmentation tools.  For my group this is where problems really start

Sent from my iPhone

> On Jul 30, 2016, at 8:41 AM, Andreas Bruckbauer <[hidden email]> wrote:
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> Dear Andrew,
>
> Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "DENNIS Andrew" <[hidden email]>
> Sent: ‎29/‎07/‎2016 23:39
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> Sorry Typo in an embarrassing part of my last message,
>
> It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "
>
> -----Original Message-----
> From: DENNIS Andrew
> Sent: 29 July 2016 23:17
> To: Confocal Microscopy List <[hidden email]>
> Subject: RE: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> Hi Esteban,
>
> I work at Andor/Bitpane so you may consider this to be a commercial response..
>
> I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?
>
> Andrew
>
>
> ________________________________________
> From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
> Sent: 29 July 2016 20:43
> To: [hidden email]
> Subject: Re: analyzing really big data sets
>
> EXTERNAL EMAIL
>
> ATTACHMENT ADVISORY
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> I just wanted to echo the praises for Arivis and MicroVolution.
>
> My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.
>
> MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
> 1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.
>
> -Esteban
>
>> On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> Would it not be much better to perform the data analysis on a scalable
>> cluster which has fast connection to the storage instead of moving
>> data around? We need to push software companies to make their
>> solutions run on these machines. Instead of buying ever bigger
>> analysis workstations which are obsolete after a few years, one would
>> just buy computing time. The cluster can be shared with bioinformatics groups.
>>
>> My take on storage is that you need to have a cheap archive, otherwise
>> there will be a point at which you run out of money to keep the ever
>> expanding storage.
>>
>> Best wishes
>>
>> Andreas
>>
>> -----Original Message-----
>> From: "Douglas Richardson" <[hidden email]>
>> Sent: ‎27/‎07/‎2016 15:34
>> To: "[hidden email]"
>> <[hidden email]>
>> Subject: Re: analyzing really big data sets
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
>> suggestion regarding Visiopharm for 2D data sets (I really love that
>> it doesn't duplicate the data into yet another proprietary file type).
>>
>>
>> However, theses are both expensive and there are open source options
>> as well.  One of our groups has a great open-source work flow for
>> imaging and registering cleared brains (imaged & registered >80
>> cleared brains, ~150TB of data). Here is the reference:
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fhcbi.fas.harvard.edu-252fpublications-252fdopamine-2Dneurons-2Dprojecting-2Dp-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DtlmNC9Erq98MzwNvJze0ir8hWU7gDI3DXqBr90y8zdU-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=CKV2L0-Cp1fHAvG9N3LtrgeBvu_ck-loIxg8iIErn8U&e=
>> osterior-striatum-form-ananatomically-distinct
>> .
>> The Tessier-Lavigne lab just released a computational method (ClearMap
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.sciencedirect.com-252fscience-252farticle-252fpii-252fS0092867416305554-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DEKbzfoCH4-252fenjdyFiwH0XaEFCbTDMITwh7xoq-252bVcyDQ-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=k_XJKfwlzqhWwv370smWTgOcWFuojKz8VJ3G7N39gOc&e=  for
>> a similar process, as has the Ueda group with their CUBIC method (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.nature.com-252fnprot-252fjournal-252fv10-252fn11-252ffull-252fnprot.2015.085.html-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DJbk4NsFW4QOz2H-252fWFoM83EVCu3ybSUCaesUtCxx58hI-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=ybKRFBy0Tt7b_lfHcBLK-szD2PRAfurqITyQXtju-ZE&e= ),
>> although these both mainly deal with ultra-microscope data which isn't
>> as intensive as other forms of lightsheet.
>>
>> Big data viewer in Fiji and Vaa3D are also good open source options
>> for viewing the data.
>>
>> On the data storage side, the above mentioned publication was done
>> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.
>> Since then, we've run 10Gbit optical fiber to all of our microscopes
>> and workstations.  Most importantly, this 10Gbit connection goes right
>> through to our expandable storage server downtown.
>>
>> I think the two big lessons we've learned are the following:
>>
>> 1) Make sure your storage is expandable, you'll never have enough.
>> We're currently at 250TB in a LUSTER configuration with plans to push
>> into PTs soon.
>> 2) You will always need to move data, make sure your connections are fast.
>> We have a 3 tier system: 1) Microscope acquisition computer > 2)
>> Processing workstations > 3) Long-term storage server.  Connections to
>> the cloud are not fast enough, so I don't feel this is an option.
>>
>> Finally, many versions of commercial microscope acquisition software
>> are unable to directly save data to network storage (or external
>> drives) no matter how fast the connection. This is a feature we need
>> to push the manufacturers for or else you'll always be limited to the
>> storage space on your acquisition computer.
>>
>> -Doug
>>
>> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
>> <[hidden email]>
>> wrote:
>>
>>> *****
>>> To join, leave or search the confocal microscopy listserv, go to:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your
>> posting.
>>> *****
>>>
>>> We have been facing the same issue since the purchase of our Zeiss
>>> Lightsheet system. On the commercial side of things, Arivis has
>>> worked
>> well
>>> for us and I would recommend giving that a shot. On the
>>> deconvolution
>> side,
>>> we recently purchased the Huygens deconvolution module and it has
>>> given
>> us
>>> nice results. We had also tested the Microvolution software and were
>> really
>>> impressed at the speed and quality of deconvolution - the price tag
>>> put
>> it
>>> out of our range for the time being, but it's definitely worth exploring.
>
>
> +++Scanned for Viruses by ForcePoint+++
>
>
> ___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dwww.oxinst.com-252femail-2Dstatement-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dre2O-252f41spaRqGYyhvCPIBjfHEfm4dsISa7Exfc04adg-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=T-jLF96F7LbpHROZuDsTfEfZlsCHlImzqWVYo9LGEJw&e=  for regulatory information.

------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================

[Remko Dijkstra]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear all,

As some of you have already pointed out: the Huygens software has a very
fast GPU (CUDA) acceleration option that allows for fast deconvolution
and is well suited for large datasets: https://svi.nl/HuygensGPU
Huygens uses an advanced bricking algorithm, that allows the
deconvolution to be performed on individual bricks that are optimally
sized for the GPU RAM. After the deconvolution, the bricks will be
merged to create the end-result. So the Huygens bricking algorithm will
handle the optimal brick sizing for you, rather than you needing to do
manual cropping (and merging) of the data.

In case of time-series and multi-channel images, it is quite
straightforward to do a frame-by-frame deconvolution. This can already
efficiently be done with the Huygens Batch Processor when the individual
frames/channels are stored as individual files. So you can process the
time frames (and channels) one-by-one rather than loading the entire
time-series or multi-channel frames entirely into RAM.

In addition, the Huygens Stitcher (https://svi.nl/Stitcher) is an
efficient 3D tile stitcher option that includes automatic vignetting
correction and allows users to apply tile-wise deconvolution before
applying the actual stitching. Every individual tile can be deconvolved
subsequently, and the stitching is performed afterwards. So for the
deconvolution of the individual tiles, much less RAM is required
compared to if the deconvolution would be done on the large (stitched)
dataset.

The Huygens 3D MIP renderer and cropping tool (based on multi-MIP), are
full multi-threaded CPU-based algorithms, so they are not limited to the
GPU RAM. Both visualization tools work on the dataset as loaded in RAM,
so no additional RAM is required for visualization. This makes both
visualization tools in Huygens very fast and the best choice for
visualizing and cropping large datasets.

Still, for many image processing methods it is important to have
sufficient RAM to store intermediate and/or end-results. Having a server
with a good amount of RAM is always beneficial or even required when
doing high-end image processing on (very) large datasets.
Fortunately, Moore's law is still increasing RAM density, putting a
server with 1TB of RAM or more within reach of most research groups
working with large datasets.

Having some additional fast SSD swap space available will allow you to
do processing even when the entire dataset does not fit entirely into
RAM. Yet this also has some limitations, since processing via swap space
is much slower compared to when the datasets fits entirely into RAM.
Computer cluster processing, or writing part of the data to disk while
processing, might be a solution for some very specific (large data)
algorithms, but for many complex image processing/analysis algorithms
this is not a practical solution.
Analysis algorithms that would work directly on the data stored on the
hard disk would be orders of magnitude slower, and would become
unnecessarily complex.

To point out some numbers when it comes to data transfer rates:
SSD's typically have a reading speed of ~500 MB/sec. To read a 1 TB
dataset from such an SSD, this takes roughly 2000 seconds (=33 minutes!)
to load it in RAM and make it available for processing.
Even when the SSD's would become much faster, they will still be limited
to the interface speed: SATA 3.2 is currently limited to ~ 2 GB/sec. So
1 TB, would then still take 500 sec to load into RAM. Current PCI-e
SSD's have a comparable transfer speed (~2 GB/s).
RAM (DDR3/4) memory bandwidth in Intel Xeon servers is around 50GB/sec
(~20 seconds for 1 TB).
The upcoming HBM 2 (high bandwidth memory) will have a throughput of
multiple hundreds of GB/s to 2 TB/s, so in the future new types of high
bandwidth memory will also significantly improve the handling of large
datasets.
With the growing interest of Big Data processing in different fields,
and with the development in virtual reality, the market for better and
faster memory and storage interface has increased significantly over the
past few years.
It is not difficult to imagine that in the near future we will be having
the option to use fast hybrid Memory/SSD storage types which will be
able to efficiently communicate with High Bandwidth Memory with high
transfer rates. This will make the processing of large data, also in
microscopy, much more efficient.

With this in mind, Scientific Volume Imaging will continue to improve
the Huygens software to handle and process large datasets efficiently.
But it also remains equally important to stay up-to-date with the latest
hardware developments and run high-end image processing software on a
system with processing capabilities that is in proportion with the file
sizes that need to be processed.

With kind regards, on behalf of the SVI team,

Remko

Huygens GPU acceleration: https://svi.nl/HuygensGPU
Fast high-quality results!
***********************************************************
Remko Dijkstra, MSc
Imaging Specialist/Account Manager
Scientific Volume Imaging bv
Tel: + 31 35 642 1626
www.svi.nl
***********************************************************
For support matters contact: [hidden email]

On 31-07-16 00:23, Cammer, Michael wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> When we do an overnight experiment looking at embryos by lightsheet, the datasets are typically in the 800GB to 1.2 TB range.  We deal with this by having each timepoint of each embryo saved as a separate czi file.  Similar for Nikon Zseries multiple color multiple position overnight timelapse; we save each position as a separate nd2 file and between timepoints we can access each position to date.  To solve this problem with the Zeiss systems, we have Zen export as individual tifs during image collection.  Micromanager also will do this.  So as long as you don't mind have to deal with 80,000 files, you can work around the problem of accessing the beginning of a sequence before the end has happened yet.
>
>
> _________________________________________
> Michael Cammer, Optical Microscopy Specialist
> http://ocs.med.nyu.edu/microscopy
> http://microscopynotes.com/
> Cell: [phone]
>
> ________________________________________
> From: Confocal Microscopy List [[hidden email]] on behalf of Andreas Bruckbauer [[hidden email]]
> Sent: Saturday, July 30, 2016 9:46 AM
> To: [hidden email]
> Subject: Re: analyzing really big data sets
>
>
> Another thing which bothers me (currently for data sets of moderate size e.g. 10 Gb) is that I first have to save the complete data set on the acquisition computer, then open it on the analysis computer and when it is fully loaded, start the analysis. It would be less time consuming if I could already start loading and analysing  the first time points while the last ones are still being saved or even not yet measured. This should be possible, but is not implemented in todays microscope systems.
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "Watkins, Simon C" <[hidden email]>
> Sent: ‎30/‎07/‎2016 10:38
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=gXXzZP8amUxJuuEYayueruKp-7Ulp0d9NTTdM0UVzJc&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=963fE12TwWxn_THDdJAbx0e9Pgcr3w9ZfDxgtJAm-_8&e=  and include the link in your posting.
> *****
>
> I am glad this question has raised such interest. Importantly though there are really two questions here.  Rendering and displaying data so that it can be visually plumbed and quantitation of structure and events within the rendered volume using potentially complex combinations of filters and segmentation tools.  For my group this is where problems really start
>
> Sent from my iPhone
>
>> On Jul 30, 2016, at 8:41 AM, Andreas Bruckbauer <[hidden email]> wrote:
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> Dear Andrew,
>>
>> Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?
>>
>> Best wishes
>>
>> Andreas
>>
>> -----Original Message-----
>> From: "DENNIS Andrew" <[hidden email]>
>> Sent: ‎29/‎07/‎2016 23:39
>> To: "[hidden email]" <[hidden email]>
>> Subject: Re: analyzing really big data sets
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> Sorry Typo in an embarrassing part of my last message,
>>
>> It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "
>>
>> -----Original Message-----
>> From: DENNIS Andrew
>> Sent: 29 July 2016 23:17
>> To: Confocal Microscopy List <[hidden email]>
>> Subject: RE: analyzing really big data sets
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> Hi Esteban,
>>
>> I work at Andor/Bitpane so you may consider this to be a commercial response..
>>
>> I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?
>>
>> Andrew
>>
>>
>> ________________________________________
>> From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
>> Sent: 29 July 2016 20:43
>> To: [hidden email]
>> Subject: Re: analyzing really big data sets
>>
>> EXTERNAL EMAIL
>>
>> ATTACHMENT ADVISORY
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> I just wanted to echo the praises for Arivis and MicroVolution.
>>
>> My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.
>>
>> MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
>> 1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.
>>
>> -Esteban
>>
>>> On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:
>>>
>>> *****
>>> To join, leave or search the confocal microscopy listserv, go to:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>>> *****
>>>
>>> Would it not be much better to perform the data analysis on a scalable
>>> cluster which has fast connection to the storage instead of moving
>>> data around? We need to push software companies to make their
>>> solutions run on these machines. Instead of buying ever bigger
>>> analysis workstations which are obsolete after a few years, one would
>>> just buy computing time. The cluster can be shared with bioinformatics groups.
>>>
>>> My take on storage is that you need to have a cheap archive, otherwise
>>> there will be a point at which you run out of money to keep the ever
>>> expanding storage.
>>>
>>> Best wishes
>>>
>>> Andreas
>>>
>>> -----Original Message-----
>>> From: "Douglas Richardson" <[hidden email]>
>>> Sent: ‎27/‎07/‎2016 15:34
>>> To: "[hidden email]"
>>> <[hidden email]>
>>> Subject: Re: analyzing really big data sets
>>>
>>> *****
>>> To join, leave or search the confocal microscopy listserv, go to:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>>> *****
>>>
>>> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
>>> suggestion regarding Visiopharm for 2D data sets (I really love that
>>> it doesn't duplicate the data into yet another proprietary file type).
>>>
>>>
>>> However, theses are both expensive and there are open source options
>>> as well.  One of our groups has a great open-source work flow for
>>> imaging and registering cleared brains (imaged & registered >80
>>> cleared brains, ~150TB of data). Here is the reference:
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fhcbi.fas.harvard.edu-252fpublications-252fdopamine-2Dneurons-2Dprojecting-2Dp-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DtlmNC9Erq98MzwNvJze0ir8hWU7gDI3DXqBr90y8zdU-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=CKV2L0-Cp1fHAvG9N3LtrgeBvu_ck-loIxg8iIErn8U&e=
>>> osterior-striatum-form-ananatomically-distinct
>>> .
>>> The Tessier-Lavigne lab just released a computational method (ClearMap
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.sciencedirect.com-252fscience-252farticle-252fpii-252fS0092867416305554-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DEKbzfoCH4-252fenjdyFiwH0XaEFCbTDMITwh7xoq-252bVcyDQ-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=k_XJKfwlzqhWwv370smWTgOcWFuojKz8VJ3G7N39gOc&e=  for
>>> a similar process, as has the Ueda group with their CUBIC method (
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.nature.com-252fnprot-252fjournal-252fv10-252fn11-252ffull-252fnprot.2015.085.html-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DJbk4NsFW4QOz2H-252fWFoM83EVCu3ybSUCaesUtCxx58hI-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=ybKRFBy0Tt7b_lfHcBLK-szD2PRAfurqITyQXtju-ZE&e= ),
>>> although these both mainly deal with ultra-microscope data which isn't
>>> as intensive as other forms of lightsheet.
>>>
>>> Big data viewer in Fiji and Vaa3D are also good open source options
>>> for viewing the data.
>>>
>>> On the data storage side, the above mentioned publication was done
>>> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.
>>> Since then, we've run 10Gbit optical fiber to all of our microscopes
>>> and workstations.  Most importantly, this 10Gbit connection goes right
>>> through to our expandable storage server downtown.
>>>
>>> I think the two big lessons we've learned are the following:
>>>
>>> 1) Make sure your storage is expandable, you'll never have enough.
>>> We're currently at 250TB in a LUSTER configuration with plans to push
>>> into PTs soon.
>>> 2) You will always need to move data, make sure your connections are fast.
>>> We have a 3 tier system: 1) Microscope acquisition computer > 2)
>>> Processing workstations > 3) Long-term storage server.  Connections to
>>> the cloud are not fast enough, so I don't feel this is an option.
>>>
>>> Finally, many versions of commercial microscope acquisition software
>>> are unable to directly save data to network storage (or external
>>> drives) no matter how fast the connection. This is a feature we need
>>> to push the manufacturers for or else you'll always be limited to the
>>> storage space on your acquisition computer.
>>>
>>> -Doug
>>>
>>> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
>>> <[hidden email]>
>>> wrote:
>>>
>>>> *****
>>>> To join, leave or search the confocal microscopy listserv, go to:
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>>>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your
>>> posting.
>>>> *****
>>>>
>>>> We have been facing the same issue since the purchase of our Zeiss
>>>> Lightsheet system. On the commercial side of things, Arivis has
>>>> worked
>>> well
>>>> for us and I would recommend giving that a shot. On the
>>>> deconvolution
>>> side,
>>>> we recently purchased the Huygens deconvolution module and it has
>>>> given
>>> us
>>>> nice results. We had also tested the Microvolution software and were
>>> really
>>>> impressed at the speed and quality of deconvolution - the price tag
>>>> put
>>> it
>>>> out of our range for the time being, but it's definitely worth exploring.
>>
>> +++Scanned for Viruses by ForcePoint+++
>>
>>
>> ___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dwww.oxinst.com-252femail-2Dstatement-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dre2O-252f41spaRqGYyhvCPIBjfHEfm4dsISa7Exfc04adg-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=T-jLF96F7LbpHROZuDsTfEfZlsCHlImzqWVYo9LGEJw&e=  for regulatory information.
> ------------------------------------------------------------
> This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
> =================================
>

[0000001ed7f52e4a-dmarc-request]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Saving the time points individually sounds good, what is then needed is to adapt automated workflows. It is simple to process all files which are currently in a directory, but more difficult to deal with files which will appear in this directory later. I think this is where some improvement in the current software is necessary.

Best wishes

Andreas

-----Original Message-----
From: "Cammer, Michael" <[hidden email]>
Sent: ‎30/‎07/‎2016 23:24
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

When we do an overnight experiment looking at embryos by lightsheet, the datasets are typically in the 800GB to 1.2 TB range.  We deal with this by having each timepoint of each embryo saved as a separate czi file.  Similar for Nikon Zseries multiple color multiple position overnight timelapse; we save each position as a separate nd2 file and between timepoints we can access each position to date.  To solve this problem with the Zeiss systems, we have Zen export as individual tifs during image collection.  Micromanager also will do this.  So as long as you don't mind have to deal with 80,000 files, you can work around the problem of accessing the beginning of a sequence before the end has happened yet.


_________________________________________
Michael Cammer, Optical Microscopy Specialist
http://ocs.med.nyu.edu/microscopy
http://microscopynotes.com/
Cell: [phone]

________________________________________
From: Confocal Microscopy List [[hidden email]] on behalf of Andreas Bruckbauer [[hidden email]]
Sent: Saturday, July 30, 2016 9:46 AM
To: [hidden email]
Subject: Re: analyzing really big data sets


Another thing which bothers me (currently for data sets of moderate size e.g. 10 Gb) is that I first have to save the complete data set on the acquisition computer, then open it on the analysis computer and when it is fully loaded, start the analysis. It would be less time consuming if I could already start loading and analysing  the first time points while the last ones are still being saved or even not yet measured. This should be possible, but is not implemented in todays microscope systems.

Best wishes

Andreas

-----Original Message-----
From: "Watkins, Simon C" <[hidden email]>
Sent: ‎30/‎07/‎2016 10:38
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=gXXzZP8amUxJuuEYayueruKp-7Ulp0d9NTTdM0UVzJc&e=
Post images on https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=963fE12TwWxn_THDdJAbx0e9Pgcr3w9ZfDxgtJAm-_8&e=  and include the link in your posting.
*****

I am glad this question has raised such interest. Importantly though there are really two questions here.  Rendering and displaying data so that it can be visually plumbed and quantitation of structure and events within the rendered volume using potentially complex combinations of filters and segmentation tools.  For my group this is where problems really start

Sent from my iPhone

> On Jul 30, 2016, at 8:41 AM, Andreas Bruckbauer <[hidden email]> wrote:
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> Dear Andrew,
>
> Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "DENNIS Andrew" <[hidden email]>
> Sent: ‎29/‎07/‎2016 23:39
> To: "[hidden email]" <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> Sorry Typo in an embarrassing part of my last message,
>
> It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "
>
> -----Original Message-----
> From: DENNIS Andrew
> Sent: 29 July 2016 23:17
> To: Confocal Microscopy List <[hidden email]>
> Subject: RE: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> Hi Esteban,
>
> I work at Andor/Bitpane so you may consider this to be a commercial response..
>
> I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?
>
> Andrew
>
>
> ________________________________________
> From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
> Sent: 29 July 2016 20:43
> To: [hidden email]
> Subject: Re: analyzing really big data sets
>
> EXTERNAL EMAIL
>
> ATTACHMENT ADVISORY
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
> *****
>
> I just wanted to echo the praises for Arivis and MicroVolution.
>
> My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.
>
> MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
> 1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.
>
> -Esteban
>
>> On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> Would it not be much better to perform the data analysis on a scalable
>> cluster which has fast connection to the storage instead of moving
>> data around? We need to push software companies to make their
>> solutions run on these machines. Instead of buying ever bigger
>> analysis workstations which are obsolete after a few years, one would
>> just buy computing time. The cluster can be shared with bioinformatics groups.
>>
>> My take on storage is that you need to have a cheap archive, otherwise
>> there will be a point at which you run out of money to keep the ever
>> expanding storage.
>>
>> Best wishes
>>
>> Andreas
>>
>> -----Original Message-----
>> From: "Douglas Richardson" <[hidden email]>
>> Sent: ‎27/‎07/‎2016 15:34
>> To: "[hidden email]"
>> <[hidden email]>
>> Subject: Re: analyzing really big data sets
>>
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252flists.umn.edu-252fcgi-2Dbin-252fwa-253fA0-253dconfocalmicroscopy-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3Dt1RGJMqvJRv0jWrfdkpO2jkIpJ7U32DZS-252bO3BGya-252buk-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=6YDQ0AiEsynZ4VSHwqLhvxAEj_ET8K1RuICrEGu5GgA&e=
>> Post images on https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-252f-252fwww.imgur.com-26data-3D01-257c01-257csimon.watkins-2540PITT.EDU-257c40add1dcc1894c2995a908d3b84cec29-257c9ef9f489e0a04eeb87cc3a526112fd0d-257c1-26sdata-3DGgH34Bo8dk1K0mfLtaKyz25wAZUABFa7Kx8Ry1bVR44-253d&d=CwIF_g&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=MVSEEplZh9F7Ve66J12Qfibcx2atQhwpw2CrzJaFMxo&s=iMGLsf1XkuzjsCaPYF25MW2T9nPSznV94o0Wl58d2k8&e=  and include the link in your posting.
>> *****
>>
>> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
>> suggestion regarding Visiopharm for 2D data sets (I really love that
>> it doesn't duplicate the data into yet another proprietary file type).
>>
>>
>> However, theses are both expensive and there are open source options
>> as well.  One of our groups has a great open-source work flow for
>> imaging and registering cleared brains (imaged & registered >80
>> cleared brains, ~150TB of data). Here is the reference:
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253a-25

[The entire original message is not included.]

[DENNIS Andrew]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Andreas,

I was working with an IMS file, so it took about 60 seconds to load the 1.2TB file and its ready for analysis straight away, I did some spots analysis and tracking.  

I think you are using an non-native format, which is loading and previewing the data and analysis isn’t possible until the full file conversion is complete.

I’d very much recommend converting the file to IMS format using the file converter and then loading into Imaris, it’s definitely a better experience. The file converter doesn't need an Imaris licence and can be run on a separate PC if you want.

You mentioned the image Database (‘Arena’), we’ve got plenty of feedback from big data users, and as a result Arena was made an optional part of the Imaris install, so you can now run Imaris without Arena.

I hope this is helpful,

Andrew

-----Original Message-----
From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Andreas Bruckbauer
Sent: 30 July 2016 08:40
To: [hidden email]
Subject: Re: analyzing really big data sets

EXTERNAL EMAIL

ATTACHMENT ADVISORY

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Andrew,

Are you sure you loaded the whole data set in 60s? My experience with Imaris is that it quickly displays a part of the data set but when you want to do any meaningful analysis (like tracking cells) it really tries to load the full dataset into memory. To analyse data sets of 10-20 GB we need a workstation with 128 GB RAM while Arivis works with very little RAM.  As I understand we are here talking about 100GB - 5TB, so loading the full dataset is wholly unpractical. Maybe something changed in recent versions of Imaris? I stopped updating since Imaris introduced this ridiculous database which fills up the local hard disk. What about using Omero instead?

Best wishes

Andreas

-----Original Message-----
From: "DENNIS Andrew" <[hidden email]>
Sent: ‎29/‎07/‎2016 23:39
To: "[hidden email]" <[hidden email]>
Subject: Re: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Sorry Typo in an embarrassing part of my last message,

It should have said " today I loaded a 1.2TB data set, it took about 60 seconds. "

-----Original Message-----
From: DENNIS Andrew
Sent: 29 July 2016 23:17
To: Confocal Microscopy List <[hidden email]>
Subject: RE: analyzing really big data sets

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Esteban,

I work at Andor/Bitpane so you may consider this to be a commercial response..

I'm interested in your comment on Imaris, today I loaded a 1.2GB data set, it took about 60 seconds. When you refer to Big data, what sizes are you talking about?

Andrew


________________________________________
From: Confocal Microscopy List [[hidden email]] on behalf of G. Esteban Fernandez [[hidden email]]
Sent: 29 July 2016 20:43
To: [hidden email]
Subject: Re: analyzing really big data sets

EXTERNAL EMAIL

ATTACHMENT ADVISORY

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I just wanted to echo the praises for Arivis and MicroVolution.

My favorite for 3D work is Imaris but it can't handle big data; when possible I downsample large datasets and work in Imaris. Amira can handle big data but in my experience it crashed more often than Arivis plus I prefer the user interface in Arivis.

MicroVolution results were comparable to Huygens and AutoQuant in my hands (qualitatively, I didn't do rigorous quantitative comparisons) in about
1/60 of the time with a lower end GPU. I mostly looked at confocal point-scanning data and didn't try truly big data. MicroVolution is limited to datasets <RAM, so you subvolume yourself before deconvolving.

-Esteban

On Jul 27, 2016 10:03 AM, "Andreas Bruckbauer" < [hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Would it not be much better to perform the data analysis on a scalable
> cluster which has fast connection to the storage instead of moving
> data around? We need to push software companies to make their
> solutions run on these machines. Instead of buying ever bigger
> analysis workstations which are obsolete after a few years, one would
> just buy computing time. The cluster can be shared with bioinformatics groups.
>
> My take on storage is that you need to have a cheap archive, otherwise
> there will be a point at which you run out of money to keep the ever
> expanding storage.
>
> Best wishes
>
> Andreas
>
> -----Original Message-----
> From: "Douglas Richardson" <[hidden email]>
> Sent: ‎27/‎07/‎2016 15:34
> To: "[hidden email]"
> <[hidden email]>
> Subject: Re: analyzing really big data sets
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> I'll echo Paul's endorsement of Arivis for 3D data sets and George's
> suggestion regarding Visiopharm for 2D data sets (I really love that
> it doesn't duplicate the data into yet another proprietary file type).
>
>
> However, theses are both expensive and there are open source options
> as well.  One of our groups has a great open-source work flow for
> imaging and registering cleared brains (imaged & registered >80
> cleared brains, ~150TB of data). Here is the reference:
>
> http://hcbi.fas.harvard.edu/publications/dopamine-neurons-projecting-p
> osterior-striatum-form-ananatomically-distinct
> .
> The Tessier-Lavigne lab just released a computational method (ClearMap
> http://www.sciencedirect.com/science/article/pii/S0092867416305554 for
> a similar process, as has the Ueda group with their CUBIC method (
> http://www.nature.com/nprot/journal/v10/n11/full/nprot.2015.085.html),
> although these both mainly deal with ultra-microscope data which isn't
> as intensive as other forms of lightsheet.
>
> Big data viewer in Fiji and Vaa3D are also good open source options
> for viewing the data.
>
> On the data storage side, the above mentioned publication was done
> mainly with a filing cabinet full of 2TB USB 3.0 external hard drives.
> Since then, we've run 10Gbit optical fiber to all of our microscopes
> and workstations.  Most importantly, this 10Gbit connection goes right
> through to our expandable storage server downtown.
>
> I think the two big lessons we've learned are the following:
>
> 1) Make sure your storage is expandable, you'll never have enough.
> We're currently at 250TB in a LUSTER configuration with plans to push
> into PTs soon.
> 2) You will always need to move data, make sure your connections are fast.
> We have a 3 tier system: 1) Microscope acquisition computer > 2)
> Processing workstations > 3) Long-term storage server.  Connections to
> the cloud are not fast enough, so I don't feel this is an option.
>
> Finally, many versions of commercial microscope acquisition software
> are unable to directly save data to network storage (or external
> drives) no matter how fast the connection. This is a feature we need
> to push the manufacturers for or else you'll always be limited to the
> storage space on your acquisition computer.
>
> -Doug
>
> On Wed, Jul 27, 2016 at 9:33 AM, Paul Paroutis
> <[hidden email]>
> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> > Post images on http://www.imgur.com and include the link in your
> posting.
> > *****
> >
> > We have been facing the same issue since the purchase of our Zeiss
> > Lightsheet system. On the commercial side of things, Arivis has
> > worked
> well
> > for us and I would recommend giving that a shot. On the
> > deconvolution
> side,
> > we recently purchased the Huygens deconvolution module and it has
> > given
> us
> > nice results. We had also tested the Microvolution software and were
> really
> > impressed at the speed and quality of deconvolution - the price tag
> > put
> it
> > out of our range for the time being, but it's definitely worth exploring.
> >
>

+++Scanned for Viruses by ForcePoint+++


___________________________________________________________________________This e-mail is confidential and is for the addressee only.  Please refer to www.oxinst.com/email-statement for regulatory information.

[Steffen Dietzel]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Am 01.08.2016 um 14:28 schrieb Remko Dijkstra:
>
> To point out some numbers when it comes to data transfer rates:
> SSD's typically have a reading speed of ~500 MB/sec. To read a 1 TB
> dataset from such an SSD, this takes roughly 2000 seconds (=33
> minutes!) to load it in RAM and make it available for processing.

Which is exactly why it should not be necessary to load the complete
data set into RAM to start working on it. Thank you for making this point.

Why do I have to wait for all color channels/time points of a data set
to be in RAM (or even worse: virtual memory) to start to
display/deconvolve/analyze the data? That is a concept that doesn't cut
it, and not for the first time in history. It seems the major advantage
of this concept is that it is easier to implement for the software
developers.

When I started with confocal microscopy in the 90ies, a 40 MByte 3D-3
Color data set was huge and we ran into similar problems. My personal
impression (based on nothing but gut feeling) is that since then, RAM
generally grew faster than our data size, so eventually the problem went
away. Until a few years ago, when fast high resolution techniques came
around (resonant scanners, 4 Mpixel cameras, light sheet..).  Now we are
back to square one, sort of. This time I do not expect the problem to go
away by itself since it appears that for most applications the 4-16
Gbyte of RAM of a standard computer is more than enough, no need to put
in more.

Steffen

--
------------------------------------------------------------
Steffen Dietzel, PD Dr. rer. nat
Ludwig-Maximilians-Universität München
Biomedical Center (BMC)
Head of the Core Facility Bioimaging

Großhaderner Straße 9
D-82152 Planegg-Martinsried
Germany

http://www.bioimaging.bmc.med.uni-muenchen.de