3D structured illumination microscopy with an axially moving illumination pattern

[Thomas Oldershaw]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear list,

I am involved in developing a three-beam 3DSIM system and have noticed that in every system I've read about, the pattern is kept fixed in relation to the objective. However, in Gustafsson et al.'s 2008 article on 3D structured illumination microscopy [1], it is stated that:

"either the axial functions I_m are also purely harmonic, or, when the three- dimensional data are acquired as a sequence of two-dimensional images with different focus, the illumination pattern is maintained fixed in relation to the focal plane of the microscope, not in relation to the object."

This is justified by showing that under this arrangement the axial illumination pattern acts to multiply through with the point spread function, effectively providing increased axial support to the overall optical transfer function 'for free' (hence why only five phase steps are required for unmixing despite seven points in the illumination structure being present).

My question relates to the situation in which the pattern remains fixed in relation to the object, but with purely harmonic axial functions I_m. Based on my limited understanding, this would seem to reduce the axial situation to that of the lateral pattern, requiring phase steps to separate out the components. Indeed, equation 4 in that paper seems to show that the axial illumination pattern does not multiply the point spread function and thereby does not give the increased axial support to the OTF 'for free'. Is this correct? If we were to build a system in which the pattern remained fixed in relation to the object would we need seven images per slice? Would we even have doubled axial resolution when we do this?

This situation seems similar to that of the multifocus SIM recently published by Abrahamsson et al. [2]. Here the authors were unable to design a 3D reconstruction routine for a similar situation in which the object remains fixed, but say that such a scheme should be possible.

Thank you for any clarity anyone can provide,
Thomas


[1] Gustafsson et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophysical Journal 94 4957-4970 (2008). https://doi.org/10.1529/biophysj.107.120345
[2] Abrahamsson et al. Multifocus structured illumination microscopy for fast volumetric super-resolution imaging. Biomedical Optics Express 8 4135-4140 (2017). https://doi.org/10.1364/BOE.8.004135

[Reto Fiolka]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Thomas

If the axial pattern is fixed to the sample, then its "sidebands" need to be actively unmixed, i.e. you need additional phase steps along the axial pattern orientation and you would need to solve the algebraic equation system and move the sidebands to their true location in Fourier space numerically.

You can think of it as the same as 2D SIM (phase stepping, unmixing, shifting), now just applied to all three dimensions. Therefore all information gets passed through the conventional, widefield 3D OTF. This means that 3D Nyquist sampling for conventional 3D microscopy is sufficient for the raw data. In normal 3D SIM with axial self modulation, you already have to apply a higher axial sampling for the final SIM resolution.

You are correct that for Sara's Multifocus SIM you would need axial phase steps to unmix the axial information components. A conventional grating based setup can not do that (it would need to shift the grating axially over large distances in image space or have a phase shifting element for the zero diffraction order). Therefore a proper 3D reconstruction was not attempted in that work.

Such a system has to my best knowledge not been realized yet, but would unlock the full potential of multifocus 3D SIM.

Reto

PS: an interesting situation occurs if you shift the axial illumination pattern at a different rate than you make z steps in a normal 3D SIM microscope. The axial information, while being the same Fourier coefficients, would be shifted to other areas in 3D Fourier space than what the normal self demodulation would do.

[Andrew York]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Seems like a good opportunity to mention SIM fusion via iterative
deconvolution. It's a very different way of thinking about the problem
compared to Mats's elegant Fourier theory, but it has the advantage that it
doesn't really care what illumination you use, so it's very easy to adapt
to different schemes, including the case of illumination that doesn't
follow the objective.

I wrote up a simple demo to show what I mean:
https://goo.gl/DKaFM3

Let me know if the code and images make sense, and let me know if you spot
any errors.

On Mon, Sep 25, 2017 at 10:45 AM, Reto Fiolka <
[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Dear Thomas
>
> If the axial pattern is fixed to the sample, then its "sidebands" need to
> be actively unmixed, i.e. you need additional phase steps along the axial
> pattern orientation and you would need to solve the algebraic equation
> system and move the sidebands to their true location in Fourier space
> numerically.
>
> You can think of it as the same as 2D SIM (phase stepping, unmixing,
> shifting), now just applied to all three dimensions. Therefore all
> information gets passed through the conventional, widefield 3D OTF. This
> means that 3D Nyquist sampling for conventional 3D microscopy is sufficient
> for the raw data. In normal 3D SIM with axial self modulation, you already
> have to apply a higher axial sampling for the final SIM resolution.
>
> You are correct that for Sara's Multifocus SIM you would need axial phase
> steps to unmix the axial information components. A conventional grating
> based setup can not do that (it would need to shift the grating axially
> over large distances in image space or have a phase shifting element for
> the zero diffraction order). Therefore a proper 3D reconstruction was not
> attempted in that work.
>
> Such a system has to my best knowledge not been realized yet, but would
> unlock the full potential of multifocus 3D SIM.
>
> Reto
>
> PS: an interesting situation occurs if you shift the axial illumination
> pattern at a different rate than you make z steps in a normal 3D SIM
> microscope. The axial information, while being the same Fourier
> coefficients, would be shifted to other areas in 3D Fourier space than what
> the normal self demodulation would do.
>

[Ian Dobbie]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Reto Fiolka <[hidden email]> writes:

> You are correct that for Sara's Multifocus SIM you would need axial
> phase steps to unmix the axial information components. A conventional
> grating based setup can not do that (it would need to shift the
> grating axially over large distances in image space or have a phase
> shifting element for the zero diffraction order). Therefore a proper
> 3D reconstruction was not attempted in that work.

In traditional 3-beam £D SIM you can phase step any of the 3 beams
relative to the other two to shift the pattern axially. If you step the
central one then you just shift the pattern axially. If you shift
either of the two outer beams you shift both axially and
laterally. On the OMX Blaze setup, you focus the stripes pattern with
the exact objective focal plane by shifting one beam relative to the
other two.

> Such a system has to my best knowledge not been realized yet, but
> would unlock the full potential of multifocus 3D SIM.

It ought to be possible to collect a data set of this type on an OMX
Blaze, although the current software will not do it. The reconstruction
would also need a bit of work as you would need to fit the starting
phase for each Z position of the pattern, or calculate how the phase
shifts as you shift the pattern in Z.  

> PS: an interesting situation occurs if you shift the axial
> illumination pattern at a different rate than you make z steps in a
> normal 3D SIM microscope. The axial information, while being the same
> Fourier coefficients, would be shifted to other areas in 3D Fourier
> space than what the normal self demodulation would do.

Isn't this in effect what the Abrahamsson multi focus device does?

Ian

[Reto Fiolka]

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Ian

"In traditional 3-beam £D SIM you can phase step any of the 3 beams
relative to the other two to shift the pattern axially. If you step the
central one then you just shift the pattern axially. If you shift
either of the two outer beams you shift both axially and
laterally. On the OMX Blaze setup, you focus the stripes pattern with
the exact objective focal plane by shifting one beam relative to the
other two."

To my knowledge it has not been attempted to forward a phase shift (say to the zero order alone) such that the 3D illumination pattern remains stationary on the sample while acquiring a 3D Stack. This is harder to achieve than keeping the axial illumination component shift invariant (i.e. fixed to the focal plane while acquiring a stack). You would need a very well characterized phase shifting element or alternatively shift the grating over large distances axially in image space.

However, the benefit of such a scheme would be that one could reduce the number of images for a 3D SIM reconstruction. Using three beam interference, one would need to unmix seven information bands per orientation, so 21 images per plane (3x7) would be the minimum for a reconstruction. However, the z-sampling of the data could be twice as coarse as in traditional 3D SIM. Thus for an equivalent z-step, one would need to compare 21 images to 30 images in conventional 3D SIM (two slices of 15 images).

This speed gain is known in the SIM community, but no one to my knowledge has dared to implement such a scheme. As mentioned, the axial phase stepping is harder to get right. If you do not keep the pattern perfectly stationary while acquiring an individual 3D stack, weird things can happen (one I mentioned at the end of my previous message).

The situation changes now with the idea of multifocus SIM, as you are forced into the 3D shift variant illumination regime where the 3D pattern is fixed to the sample while the multifocus system acquires a 3D stack in parallel. The advantages would be that you would be done with a minimum number of 21 images for the whole volume and you reconstruct twice the z slices from the raw data (the number of z-planes for multifocus is technologically limited). So it would have its merits to implement it that way.

I am not an expert in iterative deconvolution, so I leave it to Andrew York to comment what the minimum number of images would be for such an approach.


"It ought to be possible to collect a data set of this type on an OMX
Blaze, although the current software will not do it. The reconstruction
would also need a bit of work as you would need to fit the starting
phase for each Z position of the pattern, or calculate how the phase
shifts as you shift the pattern in Z.  "

That would be great if the OMX blaze could provide the axial pattern shifts with its Galvo setup.

"> PS: an interesting situation occurs if you shift the axial
> illumination pattern at a different rate than you make z steps in a
> normal 3D SIM microscope. The axial information, while being the same
> Fourier coefficients, would be shifted to other areas in 3D Fourier
> space than what the normal self demodulation would do.

Isn't this in effect what the Abrahamsson multi focus device does?"

I tried to describe a different effect, which can arise if you attempt in a conventional 3D SIM microscope setup to acquire a z-stack while trying to keep the axial illumination fixed to the sample, but you actually slowly drift away with the pattern while acquiring the stack. This lowest order error in forwarding the zero order phase has some surprising results, it can self demodulate the axial information, but place it in the wrong place of Fourier space. It could be used in some clever ways to reduce the amount of image data, but it so far has remained a curiosity in the SIM field.
This could not happen if you have instantaneous Z stacks from the Abrahamsson device, as there is no z-stepping of the objective needed.



Best,
Reto