Is Gustafsson-style SIM processing code available?

[Andrew York]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

In the papers I've read, Gustafsson et al. give excellent descriptions of
their processing algorithms, but I haven't stumbled on any actual
processing code, source or compiled, which turns raw SIM data into
superresolution images. I've rolled my own just for fun, but it would be
nice to compare to the 'gold standard'. Is this code available somewhere
that I'm missing? If not, is someone working on this, or does everyone who
builds a SIM just roll their own?

[Lutz Schaefer]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Andrew,
I do not think that there can ever be a "gold standard". As there are
components needed within the reconstruction that are essentially ill-posed,
a multitude of algorithmic solutions approaching the same problem, can be
utilized. While other methods that were not originally considered by Mats
Gustafsson, such as reconstruction in space domain (e.g. Stallinga et. al)
give algorithmic advantages within some components, the original reciprocal
space methods can't, it is hard to create a general processing pipeline that
take advantage of all variants. However, you might read from time to time
about advances on critical aspects, such as "Structured illumination
microscopy: artefact analysis and reduction utilizing a parameter
optimization approach", (L.H. Schaefer et. al, 2004) or more recently:
"Phase Optimization for structured illumination microscopy" (Kay Wicker et.
al, 2013).

For these reasons, you will find that almost everyone who wants to process
SIM data in a research context will do it on their own. There are just too
many optimization variables, forbidding a general "gold standard" solution.
For Mats classical academic implementations (Matlab, etc.) that you mention,
I am sure you can ask several authors for the code, as long as they are not
exclusively affiliated with commercial companies.

Regards
Lutz

__________________________________
L u t z S c h a e f e r
Sen. Scientist
Mathematical modeling / Computational microscopy
Advanced Imaging Methodology Consultation
16-715 Doon Village Rd.
Kitchener, ON, N2P 2A2, Canada
Phone/Fax: [phone]
Mobile: [phone]
Email: [hidden email]
Website: http://home.golden.net/~lschafer/
___________________________________

-----Original Message-----
From: Andrew York
Sent: Monday, August 12, 2013 2:27 AM
To: [hidden email]
Subject: Is Gustafsson-style SIM processing code available?

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

In the papers I've read, Gustafsson et al. give excellent descriptions of
their processing algorithms, but I haven't stumbled on any actual
processing code, source or compiled, which turns raw SIM data into
superresolution images. I've rolled my own just for fun, but it would be
nice to compare to the 'gold standard'. Is this code available somewhere
that I'm missing? If not, is someone working on this, or does everyone who
builds a SIM just roll their own?

[Alberto Diaspro]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

friends

a great sentence, in my  opinion,  comes from Giuliano Toraldo di Francia:"…After so many investigations about resolving power, one cannot escape the discouraging concusion that a very common sentence like:"The resolving power of such instrument has such a value" has no meaning. Resolving power is not a well-defined physical quantity…" (Toraldo di Francia, G. (1955) JOSA, vol.45.n.7)…
alby


On Aug 12, 2013, at 5:35 PM, Golden account <[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> *****
>
> Andrew,
> I do not think that there can ever be a "gold standard". As there are components needed within the reconstruction that are essentially ill-posed, a multitude of algorithmic solutions approaching the same problem, can be utilized. While other methods that were not originally considered by Mats Gustafsson, such as reconstruction in space domain (e.g. Stallinga et. al) give algorithmic advantages within some components, the original reciprocal space methods can't, it is hard to create a general processing pipeline that take advantage of all variants. However, you might read from time to time about advances on critical aspects, such as "Structured illumination microscopy: artefact analysis and reduction utilizing a parameter optimization approach", (L.H. Schaefer et. al, 2004) or more recently: "Phase Optimization for structured illumination microscopy" (Kay Wicker et. al, 2013).
>
> For these reasons, you will find that almost everyone who wants to process SIM data in a research context will do it on their own. There are just too many optimization variables, forbidding a general "gold standard" solution. For Mats classical academic implementations (Matlab, etc.) that you mention, I am sure you can ask several authors for the code, as long as they are not exclusively affiliated with commercial companies.
>
> Regards
> Lutz
>
> __________________________________
> L u t z S c h a e f e r
> Sen. Scientist
> Mathematical modeling / Computational microscopy
> Advanced Imaging Methodology Consultation
> 16-715 Doon Village Rd.
> Kitchener, ON, N2P 2A2, Canada
> Phone/Fax: [phone]
> Mobile: [phone]
> Email: [hidden email]
> Website: http://home.golden.net/~lschafer/
> ___________________________________
>
> -----Original Message----- From: Andrew York
> Sent: Monday, August 12, 2013 2:27 AM
> To: [hidden email]
> Subject: Is Gustafsson-style SIM processing code available?
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> *****
>
> In the papers I've read, Gustafsson et al. give excellent descriptions of
> their processing algorithms, but I haven't stumbled on any actual
> processing code, source or compiled, which turns raw SIM data into
> superresolution images. I've rolled my own just for fun, but it would be
> nice to compare to the 'gold standard'. Is this code available somewhere
> that I'm missing? If not, is someone working on this, or does everyone who
> builds a SIM just roll their own?

[Reto Fiolka]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Hi Andrew

there was a great interest by Mats and Rainer Heintzmann to have an open
source code available. Unfortunately, both have sold their software (Mats to
Applied precision and Rainer to Zeiss) and thus it was prohibited to share their
code.

Which is a bummer, both labs have spent considerable effort in optimizing their
code (runtime but also parameter fitting and best treatment in terms of error
propagation) and some of its features are quite neat and nifty.

Thus all labs that to SIM on their own have so far written their own code.

Best,
Reto

[Reto Fiolka]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

I have to disagree with the notion by Mr Schaefer that some components in
SIM processing are ill-posed.

SIM processing at its very basic is numerically very well conditioned. If one
uses a proper amount of phase steps with equal spacing over 2Pi, the
separation matrix has a low condition number and is thus numerically far from
ill-posed (e.g. singular) and can be solved by direct inversion. I remember the
condition number in 2D can be as low as 3 and that is far far away from an ill-
posed problem.

The other needed operations (shifting of the sidebands, linear filtering with the
OTF, Apodization and Fourier transforms) are also good conditioned and will not
cause generation of much numerical errors, if properly done.

Off course great efforts have been dedicated to estimate the grid parameters,
but you don't have to! In HELM (essentially SIM) in the group of Prof. Stemmer,
we always directly imaged the illumination pattern coherently and measured the
interference pattern orientation, period and phase steps precisely. Then you do
not need to estimate them!

Of most importance for an artefact free reconstruction is the orientation and
precise period of the grid pattern, but this can be measured accurately.

Even in 3D SIM these experimental parameters can be directly measured, as
the coarse pattern can be imaged with a thin fluorescent film. The fine pattern
is exactly at twice the frequency of the coarse pattern.

As we have shown in the recent paper by Wicker et al on parameter
optimization, phase step errors themselves are not that severe, thus in my
opinion you do not necessarily need to optimize them. The grid spacing remains
fixed in a grating setup and its orientation is reproducible, so such parameters
you don't need to estimate, you simply measure them once.

Best,
Reto

[Ian Dobbie]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Reto Fiolka <[hidden email]> writes:

> Hi Andrew
>
> there was a great interest by Mats and Rainer Heintzmann to have an open
> source code available. Unfortunately, both have sold their software (Mats to
> Applied precision and Rainer to Zeiss) and thus it was prohibited to share their
> code.
>
> Which is a bummer, both labs have spent considerable effort in optimizing their
> code (runtime but also parameter fitting and best treatment in terms of error
> propagation) and some of its features are quite neat and nifty.
>
> Thus all labs that to SIM on their own have so far written their own code.

Maybe we should take this as a call to action to provide something open
source. Even it was not extensively optimised for speed, an open source
implementation would allow people to check their reconstruction code
against a known good source.

It seems as though Andrew has working implementation, might that be a
place to start? Andrew I don't know if you might be willing to release
the code under some open source license and post it somewhere (github or
sourceforge maybe?).


Ian
--

[Lutz Schaefer]

Reto,
strictly, the inverse filtering is always ill posed and therefore needs regularization. This is mostly done with simple Tikhonov but requires a regularization parameter to be adjusted according to the Gaussian noise content of your measurements. In fluorescence microscopy mostly Poisson photon generation prevails, limiting the usefulness of such simple regularized inverse filter. Further, the correct fusion of the shifted Fourier components in discrete space is nontrivial, e.g. can be done only approximatlely especially under presence of aberrations. Finally, the necessity of the always used apodization before inverse transform indicates yet another form of regularization to prevent artefacts due to incorrect fusion. No matter how its done, this heuristic apodization cuts off higher spatial frequencies. I do not want to comment on your choice of grating shifts, but for certain interests they dont need to be equidistant within 2pi.
In my original post, I just wanted to raise awareness of the inherent non triviality and that there cant be a "gold standard" as some aspects are still not fully understood and therefore the potential for improvement exists. 

Hope that helps for some clarification.
Regards
Lutz

Sent from Samsung Mobile

-------- Original message --------
Subject: Re: Is Gustafsson-style SIM processing code available?
From: Reto Fiolka <[hidden email]>
To: [hidden email]
CC:  

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

I have to disagree with the notion by Mr Schaefer that some components in
SIM processing are ill-posed.

SIM processing at its very basic is numerically very well conditioned. If one
uses a proper amount of phase steps with equal spacing over 2Pi, the
separation matrix has a low condition number and is thus numerically far from
ill-posed (e.g. singular) and can be solved by direct inversion. I remember the
condition number in 2D can be as low as 3 and that is far far away from an ill-
posed problem.

The other needed operations (shifting of the sidebands, linear filtering with the
OTF, Apodization and Fourier transforms) are also good conditioned and will not
cause generation of much numerical errors, if properly done.

Off course great efforts have been dedicated to estimate the grid parameters,
but you don't have to! In HELM (essentially SIM) in the group of Prof. Stemmer,
we always directly imaged the illumination pattern coherently and measured the
interference pattern orientation, period and phase steps precisely. Then you do
not need to estimate them!

Of most importance for an artefact free reconstruction is the orientation and
precise period of the grid pattern, but this can be measured accurately.

Even in 3D SIM these experimental parameters can be directly measured, as
the coarse pattern can be imaged with a thin fluorescent film. The fine pattern
is exactly at twice the frequency of the coarse pattern.

As we have shown in the recent paper by Wicker et al on parameter
optimization, phase step errors themselves are not that severe, thus in my
opinion you do not necessarily need to optimize them. The grid spacing remains
fixed in a grating setup and its orientation is reproducible, so such parameters
you don't need to estimate, you simply measure them once.

Best,
Reto

[Reto Fiolka]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Hello Lutz

I have worked with Rainer Heintzmann and Mats Gustafsson for some time on
SIM and I can tell you from a practical perspective that SIM can be made to
work well even without Thikonov regularization or any iterative optimization
scheme. We compared the reconstructions to a ground truth (e.g. AFM image
of beads) and the results where correct and not bogus. Thus while you are
right that there are many improvements possible in the reconstruction, SIM
data, if acquired properly, can be reconstructed successfully with a pretty
simple, linear, non-iterative algorithm. I am insisting on this since I don't want
that people are scared off by the post-processing.

Working with Mats, we never had the need to use any regularization
in SIM, it was always just linear (one step, non iterative) deconvolution
assisted with a Wiener filter (with the Wiener factor inversely proportional to
the signal-to-noise ratio ) . In the fusion of the components, in the overlap
region a weighted average of the two components according to their signal
strength was taken. This made any regularization assisted deconvolution seem
unnecessary, since some low SNR regions got substituted by components with
high signal strength. The use of a Wiener filter is crude and there is no doubt
that there are better statistical measures and tricks, but it worked and it is
simple.

Since the inner borders of each spectral component got fixed by the overlap, in
my opinion only the very border of the reconstructed passband might benefit
from regularization assisted reconstruction, but this would not result in much
resolution gain.

The necessity for an apodization comes from the fact that the reconstructed
spectrum has essentially an envelope of a tophat function, which will cause
ringing, but would in theory have the best transfer of even the highest
frequency components. The choice of apodization is a tradeoff between
ringing/undershoots and resolution. Once you have come to a reconstruction of
the Fourier components and restored a uniform transfer function within a
certain support, this is inevitable.

You are right that you can chose your phase steps as you please, however the
condition number will increase and in worst case the system becomes singular.
Equidistant phase steps result in the lowest condition number. You can of
course make more phase steps than necessary and solve the over-determined
equation system with a pseudo inverse or other methods. Is this the situation
where you consider non equidistant phase steps better?

All I wanted to say is that a properly built SIM system (which involves care
about drift of components and removing system aberrations) that is operated
on not to thick samples (i.e. single cells) worked great in our hands without any
constrained iterative reconstruction or other complicated algorithms that might
or might not converge to a meaningful solution. It was a pretty deterministic
system where there was no need to fumble with reconstruction parameters or
anything in daily use once it was properly setup.

Off course I appreciate your and other groups efforts to make SIM work in non
ideal conditions. This may include thick samples where you have aberrations
and the shot noise of the out of focus blur becomes dominant. Or TIRF SIM,
where the overlap region of the spectra becomes very limited and noisy. Or
nonlinear SIM where the higher harmonics become really weak.

Best,
Reto

[Lutz Schaefer]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Dear Reto,
appreciating your latest contribution (with Kai, Ondrej and Rainer), I have
been working on inverse problems over the last 20+ years and regarding SIM
have been in contact with Rainer and the late Mats Gustafsson since about
2001. I mainly agree with your reply, however let me correct you in what
appears to me as misunderstandings:

> SIM can be made to work well
> even without Thikonov regularization
>
As said previously, you need regularization to approximate a numerical
solution to the inverse problem. This has nothing to do how you want solve
this problem (e.g. linear, nonlinear, iteratively... etc.). It is important
to understand this fact, else your solution will be instable.

> it was always just linear (one step, non iterative)
> deconvolution assisted with a Wiener filter
>
Here is what I believe the main part of your confusion. The 'generalized'
Wiener filter is identical to solving the inverse problem with a Tikhonov
regularized inverse filter (equation 5 in your paper).

> (with the Wiener factor inversely proportional to
> the signal-to-noise ratio )
>
This quantity w in the denominator is in fact identical to the weight
between data fit (e.g. likelihood function) and regularization term for the
special case of Tikhonov regularization. The data fit term is in this case
the Gaussian likelihood, -or residuum, so that the sufficiently general
restoration functional:
    Likelihood + w Regularization approaches a minimum.
It can be shown with a simple partial derivative, for the linear
Gauss-likelihood, a direct solution, identical to the 'generalized Wiener
filter' results in Fourier space. For more details I think we should carry
on this conversation off the list...

Now to your choice of the regularization parameter w>=0: The approximate(!)
relation to the SNR is also known as the "discrepancy method" in the older
astronomy literature. This is not necessarily always optimal. Despite the
approximate nature of the reciprocal proportionality, the SNR is often not
known. Also, the OTF and the observation itself play a role in the
determination of your w to satisfy the restoration functional. Often this
quantity is found by trial and error. I personally favor a generalized cross
validation, which does not always guarantee visually pleasing results.
Conclusion, you cannot state (e.g. your last post) that you work without
regularization (w=0) as this would result in instability.

Regarding apodization:
> Once you have come to a reconstruction of the
> Fourier components and restored a uniform
> transfer function within a certain support, this is inevitable.
>
I see a limitation in this method suppressing high spatial frequencies, that
we actually want to see in the result. As you say, due to the introduction
of discontinuities due to fusion, it seems inevitable (in one form or
another) in Fourier space, but what about reconstruction in space domain?

Regarding phase shifts:
> You can of course make more phase steps than necessary and solve the
> over-determined
> equation system with a pseudo inverse or other methods. Is this the
> situation
> where you consider non equidistant phase steps better?
>
You are certainly correct, the condition number is lowest in the trivial,
equidistant case. We are using non-equidistant steps to suppress residual
stripe artefacts that arise from non-sinusoidal gratings for the incoherent
SIM case. We also often gather more data and solve the over determined
system of equations via least squares or other suitable methods.

> All I wanted to say is that a properly built SIM system (which involves
> care
> about drift of components and removing system aberrations) that is
> operated
> on not to thick samples (i.e. single cells) worked great in our hands
> without any
>
I do understand your point that for your samples you were successful most of
the time. However, I would be a bit cautious about generalizations thereof.

Regards
Lutz

__________________________________
L u t z S c h a e f e r
Sen. Scientist
Mathematical modeling / Computational microscopy
Advanced Imaging Methodology Consultation
16-715 Doon Village Rd.
Kitchener, ON, N2P 2A2, Canada
Phone/Fax: [phone]
Mobile: [phone]
Email: [hidden email]

___________________________________
-----Original Message-----
From: Reto Fiolka
Sent: Tuesday, August 13, 2013 1:04 PM
To: [hidden email]
Subject: Re: Is Gustafsson-style SIM processing code available?
*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Hello Lutz

I have worked with Rainer Heintzmann and Mats Gustafsson for some time on
SIM and I can tell you from a practical perspective that SIM can be made to
work well even without Thikonov regularization or any iterative optimization
scheme. We compared the reconstructions to a ground truth (e.g. AFM image
of beads) and the results where correct and not bogus. Thus while you are
right that there are many improvements possible in the reconstruction, SIM
data, if acquired properly, can be reconstructed successfully with a pretty
simple, linear, non-iterative algorithm. I am insisting on this since I
don't want
that people are scared off by the post-processing.

Working with Mats, we never had the need to use any regularization
in SIM, it was always just linear (one step, non iterative) deconvolution
assisted with a Wiener filter (with the Wiener factor inversely proportional
to
the signal-to-noise ratio ) . In the fusion of the components, in the
overlap
region a weighted average of the two components according to their signal
strength was taken. This made any regularization assisted deconvolution seem
unnecessary, since some low SNR regions got substituted by components with
high signal strength. The use of a Wiener filter is crude and there is no
doubt
that there are better statistical measures and tricks, but it worked and it
is
simple.

Since the inner borders of each spectral component got fixed by the overlap,
in
my opinion only the very border of the reconstructed passband might benefit
from regularization assisted reconstruction, but this would not result in
much
resolution gain.

The necessity for an apodization comes from the fact that the reconstructed
spectrum has essentially an envelope of a tophat function, which will cause
ringing, but would in theory have the best transfer of even the highest
frequency components. The choice of apodization is a tradeoff between
ringing/undershoots and resolution. Once you have come to a reconstruction
of
the Fourier components and restored a uniform transfer function within a
certain support, this is inevitable.

You are right that you can chose your phase steps as you please, however the
condition number will increase and in worst case the system becomes
singular.
Equidistant phase steps result in the lowest condition number. You can of
course make more phase steps than necessary and solve the over-determined
equation system with a pseudo inverse or other methods. Is this the
situation
where you consider non equidistant phase steps better?

All I wanted to say is that a properly built SIM system (which involves care
about drift of components and removing system aberrations) that is operated
on not to thick samples (i.e. single cells) worked great in our hands
without any
constrained iterative reconstruction or other complicated algorithms that
might
or might not converge to a meaningful solution. It was a pretty
deterministic
system where there was no need to fumble with reconstruction parameters or
anything in daily use once it was properly setup.

Off course I appreciate your and other groups efforts to make SIM work in
non
ideal conditions. This may include thick samples where you have aberrations
and the shot noise of the out of focus blur becomes dominant. Or TIRF SIM,
where the overlap region of the spectra becomes very limited and noisy. Or
nonlinear SIM where the higher harmonics become really weak.

Best,
Reto