Deep Learning for microscopy image restoration... with CARE!

[Florian Jug]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi microscophiles,

for the once who need to squeeze a bit more out of their fluorescence microscopy data: https://www.biorxiv.org/content/early/2017/12/19/236463

You can also start by seeing some examples on our website:
http://csbdeep.bioimagecomputing.com

Or let us know what you think on Twitter:
https://twitter.com/florianjug/status/943531544148357121

Fiji and KNIME integration for all examples exist (tutorial also on the website).
Training code will be available soon.

Merry Christmas to all of you,
Florian (and Martin, Uwe, Ricardo, PAvel, Loic, and Gene)

[Ricardo Henriques]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Thank you for the outstanding work Florian.
Can't stop looking at the movies:
http://csbdeep.bioimagecomputing.com/videos.html
This stuff is truly amazing!

On Thu, 21 Dec 2017 at 11:18 Florian Jug <[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Hi microscophiles,
>
> for the once who need to squeeze a bit more out of their fluorescence
> microscopy data: https://www.biorxiv.org/content/early/2017/12/19/236463
>
> You can also start by seeing some examples on our website:
> http://csbdeep.bioimagecomputing.com
>
> Or let us know what you think on Twitter:
> https://twitter.com/florianjug/status/943531544148357121
>
> Fiji and KNIME integration for all examples exist (tutorial also on the
> website).
> Training code will be available soon.
>
> Merry Christmas to all of you,
> Florian (and Martin, Uwe, Ricardo, PAvel, Loic, and Gene)

--
Ricardo Henriques, Associate Professor
<http://www.ucl.ac.uk/lmcb/research-group/ricardo-henriques-research-group>
MRC-Laboratory for Molecular Cell Biology
University College London
Gower Street, London WC1E 6BT, UK
Twitter: @HenriquesLab <https://twitter.com/HenriquesLab>

[0000001ed7f52e4a-dmarc-request]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****



These are indeed fascinating results, but also very bold claims: "CARE networks can enhance widefield images to a resolution usually only
obtainable with super-resolution microscopy, yet at considerably higher frame rates." I guess it works by identifying structures it has previously learned and puts them in the right place? While this seems to work well with your samples, I wonder how prone to artefacts this method is, when applied to the "wrong" dataset, e.g. one which it is not trained on? When do I have to train the method on new ground-truth and when can I just go ahead and use it? Probably all questions for further papers. Meanwhile I would recommend to add a scale to the line plots in Figure 4 and scale bars to the zoomed-in images, otherwise the claim for super-resolution cannot be verified (I could not find the supporting figures). Good luck with this and please update us on future work! I hope a lot of image analysis problems will be solved by AI in the future.

best wishes

Andreas





-----Original Message-----
From: Florian Jug <[hidden email]>
To: CONFOCALMICROSCOPY <[hidden email]>
Sent: Thu, 21 Dec 2017 11:18
Subject: Deep Learning for microscopy image restoration... with CARE!

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi microscophiles,

for the once who need to squeeze a bit more out of their fluorescence microscopy data: https://www.biorxiv.org/content/early/2017/12/19/236463

You can also start by seeing some examples on our website:
http://csbdeep.bioimagecomputing.com

Or let us know what you think on Twitter:
https://twitter.com/florianjug/status/943531544148357121

Fiji and KNIME integration for all examples exist (tutorial also on the website).
Training code will be available soon.

Merry Christmas to all of you,
Florian (and Martin, Uwe, Ricardo, PAvel, Loic, and Gene)

[Kirti Prakash]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Really impressive, exciting work and I love the open source spirit.

Some comments:
To show the resolution improvement, I would take a nice diffraction-limited structure like synaptonemal complex (SC). It has two strands, 150 nm apart, not resolvable in widefield but with superres.
SC is also a suitable model structure to test the training/learning algorithms as we (more or less) know the ground truth (topology+distance). MTs have pseudo-2D topology, and unless one is hitting 20 nm resolution, it is not the best model structure for superres.

Seconding to Andreas, though one can guess the scale bars, I would add them to both images and plots.  

A few important contributions in the similar direction:
Image Super-Resolution Using Deep Convolutional Networks
https://arxiv.org/pdf/1501.00092.pdf
Deep learning microscopy
https://www.osapublishing.org/optica/abstract.cfm?uri=optica-4-11-1437

And the classic one on denoising:
http://www.pnas.org/content/107/37/16016.abstract

Best wishes,
Kirti

[Ricardo Henriques]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Thank you Andreas. Indeed you are right... If you train the network to only
see microtubules, that's what it will try to see, even when you give it
another structure. So, artefacts are very possible if the user doesn't do
things properly - as it's true with any analytical approach. I hope the
paper is explicit in this aspect though.

There are some interesting points regarding this... For example, what if
you train for the same structure, seen at the same scale, but the images
used have completely different noise properties. These are very much
questions to exploit 😊.

We'll make sure regarding scale bars.

All the best,
-Ricardo

On Thu, 21 Dec 2017, 21:56 Andreas Bruckbauer, <
[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
>
>
> These are indeed fascinating results, but also very bold claims: "CARE
> networks can enhance widefield images to a resolution usually only
> obtainable with super-resolution microscopy, yet at considerably higher
> frame rates." I guess it works by identifying structures it has previously
> learned and puts them in the right place? While this seems to work well
> with your samples, I wonder how prone to artefacts this method is, when
> applied to the "wrong" dataset, e.g. one which it is not trained on? When
> do I have to train the method on new ground-truth and when can I just go
> ahead and use it? Probably all questions for further papers. Meanwhile I
> would recommend to add a scale to the line plots in Figure 4 and scale bars
> to the zoomed-in images, otherwise the claim for super-resolution cannot be
> verified (I could not find the supporting figures). Good luck with this and
> please update us on future work! I hope a lot of image analysis problems
> will be solved by AI in the future.
>
> best wishes
>
> Andreas
>
>
>
>
>
> -----Original Message-----
> From: Florian Jug <[hidden email]>
> To: CONFOCALMICROSCOPY <[hidden email]>
> Sent: Thu, 21 Dec 2017 11:18
> Subject: Deep Learning for microscopy image restoration... with CARE!
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Hi microscophiles,
>
> for the once who need to squeeze a bit more out of their fluorescence
> microscopy data: https://www.biorxiv.org/content/early/2017/12/19/236463
>
> You can also start by seeing some examples on our website:
> http://csbdeep.bioimagecomputing.com
>
> Or let us know what you think on Twitter:
> https://twitter.com/florianjug/status/943531544148357121
>
> Fiji and KNIME integration for all examples exist (tutorial also on the
> website).
> Training code will be available soon.
>
> Merry Christmas to all of you,
> Florian (and Martin, Uwe, Ricardo, PAvel, Loic, and Gene)
>

--
Ricardo Henriques, Associate Professor
<http://www.ucl.ac.uk/lmcb/research-group/ricardo-henriques-research-group>
MRC-Laboratory for Molecular Cell Biology
University College London
Gower Street, London WC1E 6BT, UK
Twitter: @HenriquesLab <https://twitter.com/HenriquesLab>

[Ricardo Henriques]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi Kirti,

Indeed SC would be very nice structures to show.

I'm unsure about your views about microtubules, particularly given their
use as the gold standard structures to show SR. You frequently get
microtubule convergence or crossings that convey complex structure that are
good challenges for such algorithms.

Thank you for the advice.

All the best,

-Ricardo

On Sat, 23 Dec 2017, 19:57 Kirti Prakash, <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Really impressive, exciting work and I love the open source spirit.
>
> Some comments:
> To show the resolution improvement, I would take a nice
> diffraction-limited structure like synaptonemal complex (SC). It has two
> strands, 150 nm apart, not resolvable in widefield but with superres.
> SC is also a suitable model structure to test the training/learning
> algorithms as we (more or less) know the ground truth (topology+distance).
> MTs have pseudo-2D topology, and unless one is hitting 20 nm resolution, it
> is not the best model structure for superres.
>
> Seconding to Andreas, though one can guess the scale bars, I would add
> them to both images and plots.
>
> A few important contributions in the similar direction:
> Image Super-Resolution Using Deep Convolutional Networks
> https://arxiv.org/pdf/1501.00092.pdf
> Deep learning microscopy
> https://www.osapublishing.org/optica/abstract.cfm?uri=optica-4-11-1437
>
> And the classic one on denoising:
> http://www.pnas.org/content/107/37/16016.abstract
>
> Best wishes,
> Kirti
>

--
Ricardo Henriques, Associate Professor
<http://www.ucl.ac.uk/lmcb/research-group/ricardo-henriques-research-group>
MRC-Laboratory for Molecular Cell Biology
University College London
Gower Street, London WC1E 6BT, UK
Twitter: @HenriquesLab <https://twitter.com/HenriquesLab>