Confocal Microscopy List - Re: Digital imaging ethics as pertaining to the enhancement of microscopy images with artificial intelligence

Re: Digital imaging ethics as pertaining to the enhancement of microscopy images with artificial intelligence

Posted by Cromey, Douglas W - (dcromey) on
URL: http://confocal-microscopy-list.275.s1.nabble.com/Digital-imaging-ethics-as-pertaining-to-the-enhancement-of-microscopy-images-with-artificial-intellie-tp7588915p7588935.html

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Sylvie,

Kirsten Miles (and colleagues) looked into this a while back and I don't think anything has changed. Most of the software we use for manipulating digital images is terrible at creating an "audit trail" of what was done to an image.

In Photoshop you have to remember to turn the feature on (EDIT | PREFERENCES | HISTORY LOG), and I am pretty sure that is the case every time you start up the Photoshop software (i.e., it's not a persistent setting). And, it doesn't concatenate the information across multiple editing sessions.

Last known ImageJ could sort of create a text file, if you knew how to ask for it.

Paul Thompson was working on developing a feature for audit trails in GIMP, but it sounded like there were some issues in the plugin capturing all of the data you would want in an audit trail.

I tell my users that they are most likely going to need to record their steps "by hand".

Doug

------------------------------------------------------------------------------------------
Douglas W. Cromey, M.S. - Associate Scientific Investigator
Dept. of Cellular & Molecular Medicine, University of Arizona
1501 N. Campbell Ave, Tucson, AZ 85724-5044 USA

office: LSN 463 email: [hidden email]
voice: 520-626-2824 fax: 520-626-2097

http://microscopy.arizona.edu/learn/microscopy-imaging-resources-www
Home of: "Microscopy and Imaging Resources on the WWW"

UA Microscopy Alliance - http://microscopy.arizona.edu

-----Original Message-----
From: Confocal Microscopy List <[hidden email]> On Behalf Of Sylvie Le Guyader
Sent: Tuesday, November 20, 2018 10:46 AM
To: [hidden email]
Subject: Re: Digital imaging ethics as pertaining to the enhancement of microscopy images with artificial intelligence

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

'users can do just about anything to a digital image, as long as they provide a detailed protocol of what was done to the published image (and, as was pointed out, they retain the original unaltered raw data). (...) if the protocol is out there, they might be called an idiot based on their image processing protocol, but they cannot be called a cheat.'

Thanks Doug! I totally agree and will use your 'guideline' in the future. :) Clearly if there is something a user does to an image that they would rather not write in the protocol, it is best they don't do it!

In the era of metadata, I wonder why there is no way yet to automatically add to an image metadata what type of processing was done after acquisition. Aren't processing commands very small text files? I am not talking about saving layers like in Photoshop so one can Undo but simply recording the commands. That would allow researchers to recall the pipeline they used 1 year before and also allow others to reproduce the same type of pipeline with a different software. A bit like an embedded Fiji macro recording. :)

Med vänlig hälsning / Best regards

Sylvie

@@@@@@@@@@@@@@@@@@@@@@@@
Sylvie Le Guyader, PhD
Live Cell Imaging Facility Manager
Karolinska Institutet- Bionut Dpt
Hälsovägen 7C,
Room 7362 (lab)/7840 (office)
14157 Huddinge, Sweden
mobile: +46 (0) 73 733 5008
LCI website
Follow our microscopy blog!

-----Original Message-----
From: Confocal Microscopy List <[hidden email]> On Behalf Of Cromey, Douglas W - (dcromey)
Sent: den 20 november 2018 17:45
To: [hidden email]
Subject: Re: Digital imaging ethics as pertaining to the enhancement of microscopy images with artificial intelligence

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Wow. Fascinating discussions! John - I went to a small meeting last fall (BioImage Informatics) where one of the presenters was either from the group that published the paper you cited (https://www.nature.com/articles/s41592-018-0194-9) or did something similar. It was a fascinating presentation and it sure seemed convincing (after using a multi-channel training set, the AI was able to predict three colors of fluorescence from just a greyscale transmitted light image). At the end of the talk I raised my hand, admitted that I was much more of a microscopist than a computer scientist, and asked "what happens when the biology turns around and 'bites you in the butt'?" The presenter sputtered a bit about doing good controls, but as he was someone who was probably more of a computer scientist, I don't think he grasped my point. At lunch the next day several others who were more microscopy oriented agreed with me, we have all learned the hard way that biological samples (especially live samples) can occasionally do things that are completely unexpected and very hard to explain. Having read a bit on the topic of AI, it seems like (as others have noted here) in carefully picked use cases, AI/Deep Learning/Neural Networks have been able to do amazing things with image processing and recognition. Some of the problems within the field seem to be code written in different programming languages (hard to share and/or compare), labs unwilling to publish or share their code, etc. (http://science.sciencemag.org/content/359/6377/725.long)

One thing to cheer about with AI is the application of algorithms to the task of culling through the literature. This seems more promising, in the short term, than some of the image processing/recognition/manipulation algorithms.

About those image ethics guidelines... Back in 1997 our local MSA affiliated society held a panel discussion to ask some local "experts" about working with digital images, since this was a new area for most of us. The panel's answers to our questions were all over the map, and some were along the lines of "trust me, I'm a scientist". The most telling, for me, was our local biomedical photographer sharing with the audience (back when Photoshop was only version 3.0) that his clients were already asking for gel bands to be selectively enhanced, or incorrectly manipulated. Since my job is to support investigators at an NIH funded center, I figured I needed to know how to explain what was appropriate to my users (especially since I was just getting into the field myself), so I set out to find some guidelines. When I didn't find them online or in the literature, I set out to learn more about the topic (a big tip of the hat to Dr. John Russ, and many others) and eventually wrote the guidelines for a local newsletter, and then published them on a webpage and ultimately was asked to turn a conference talk into a paper.

It's always been my contention that users can do just about anything to a digital image, as long as they provide a detailed protocol of what was done to the published image (and, as was pointed out, they retain the original unaltered raw data). Since the HHS/NIH/ORI definition of misconduct includes the important caveat that "honest error" and "differences of scientific opinion" do not constitute misconduct, I tell users that if the protocol is out there, they might be called an idiot based on their image processing protocol, but they cannot be called a cheat. With AI image manipulation, I think it would be much more difficult to explain all the steps, especially since (as noted in one of the YouTube videos cited earlier) the computer scientists don't always completely understand what is happening to get the end result. Also (said somewhat tongue-in-cheek), if the AI software uses what it knows about or actual parts of other images to manipulate and create a new image, isn't that a bit like plagiarism?

Probably more timely for me than the future issues with AI is helping our users understand how to appropriately interpret data from current generation microscopes (SIM, Airyscan, etc) that has been processed. I'm still on this learning curve (Fourier math is cool and complicated!), trying to get to the point where I can be a resource in this new field.

As always, I am very grateful for the shared wisdom and experiences on this listserv.
Doug

------------------------------------------------------------------------------------------
Douglas W. Cromey, M.S. - Associate Scientific Investigator Dept. of Cellular & Molecular Medicine, University of Arizona
1501 N. Campbell Ave, Tucson, AZ 85724-5044 USA

office: LSN 463 email: [hidden email]
voice: 520-626-2824 fax: 520-626-2097

http://microscopy.arizona.edu/learn/microscopy-imaging-resources-www
Home of: "Microscopy and Imaging Resources on the WWW"

UA Microscopy Alliance - http://microscopy.arizona.edu

-----Original Message-----
From: Confocal Microscopy List <[hidden email]> On Behalf Of Oshel, Philip Eugene
Sent: Tuesday, November 20, 2018 6:11 AM
To: [hidden email]
Subject: Re: Digital imaging ethics as pertaining to the enhancement of microscopy images with artificial intelligence

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Good reference, thanks!

Note: I would like to publish an article (or two) on this issue in Microscopy Today. If someone has something they'd like to contribute, please contact me.

Phil
P.S. The clip was from Futurama.
-------------
Philip Oshel
Technical Editor, Microscopy Today
Imaging Facility Director
Biology Department
1304 Biosciences
1455 Calumet Ct.
Central Michigan University
Mt. Pleasant, MI 48859
(989) 774-3576
www(dot)microscopy-today(dot)com

-----Original Message-----
From: Confocal Microscopy List <[hidden email]> on behalf of Jason Swedlow <[hidden email]>
Reply-To: Confocal Microscopy List <[hidden email]>
Date: Monday, 19November, 2018 at 19:16
To: "[hidden email]" <[hidden email]>
Subject: Re: Digital imaging ethics as pertaining to the enhancement of microscopy images with artificial intelligence

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Hi All-

On a more serious note (although any comment that cites The Simpsons is an
excellent one), see https://www.nature.com/articles/s41592-018-0195-8 for
an update and proposal for image data publication resources.

Cheers,

Jason

On Mon, Nov 19, 2018 at 7:25 PM Oshel, Philip Eugene <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> John,
>
> You forgot this clip:
> https://www.youtube.com/watch?v=WwnI0RS6J5A
>
> Phil
> -------------
> Philip Oshel
> Imaging Facility Director
> Biology Department
> 1304 Biosciences
> 1455 Calumet Ct.
> Central Michigan University
> Mt. Pleasant, MI 48859
> 989 774-3576 office
> 989 774-7567 lab
>
> -----Original Message-----
> From: Confocal Microscopy List <[hidden email]> on
> behalf of John Oreopoulos <[hidden email]>
> Reply-To: Confocal Microscopy List <[hidden email]>
> Date: Friday, 16November, 2018 at 21:33
> To: "[hidden email]" <[hidden email]>
> Subject: Digital imaging ethics as pertaining to the enhancement of
> microscopy images with artificial intelligence
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your
> posting.
> *****
>
> Earlier today a few people (including myself) brought up Doug Cromey's
> excellent treatise on digital imaging ethics in a related thread that dealt
> with training new microscope users within a research setting. Lately I've
> been hearing a lot about applications of machine learning and artificial
> intelligence to "improve", "de-noise", or "fix" images (microscopy or
> otherwise), extracting new information from low-resolution images, and even
> creating new 3D views of samples with very little information. Here is just
> one such example from Nvidia and MIT:
>
>
> https://news.developer.nvidia.com/ai-can-now-fix-your-grainy-photos-by-only-looking-at-grainy-photos/
>
> https://www.youtube.com/watch?time_continue=84&v=pp7HdI0-MIo
>
> It's clear that the microscopy world will eventually come to a head
> with this technology. I think I've seen a few research articles on this
> topic now, and this month's issue of Nature Methods has a paper on this
> topic too:
>
> https://www.nature.com/articles/s41592-018-0194-9
>
> I've been wondering if and how Cromey's guide for digital imaging
> ethics should be altered when it comes to AI-assisted microscope imaging.
> Should it be allowed/accepted? Other readings of mine on AI show that
> machine learning algorithms can produce biased results if the training
> datasets are incomplete in some way, and the very nature of machine
> learning makes it difficult to understand why it produced a certain result,
> since the deep learning neural networks that are used to generate the
> results are essentially black boxes that can't easily be probed. But on the
> other hand, I'm constantly blown away by what I've seen so far online for
> other various applications of AI (facial recognition, translation, etc.).
>
> I also just finished a good read about AI from the perspective of
> economics:
>
> https://www.predictionmachines.ai/
>
> https://youtu.be/5G0PbwtiMJk
>
> The basic message of this book is that AI makes prediction cheap. When
> something is cheap, we use more of it. Other processes that complement
> prediction, like judgement (by a human or otherwise) becomes more valuable.
> It's easy to see how the lessons of this book could be re-framed for
> imaging science.
>
> Curious to know the community's opinion on this matter. I used to
> laugh at the following video, but now I'm not laughing:
>
> https://www.youtube.com/watch?v=LhF_56SxrGk
>
> John Oreopoulos
>
>
>

--
**************************
Centre for Gene Regulation & Expression
School of Life Sciences
University of Dundee
Dundee DD1 5EH
United Kingdom

phone (01382) 385819
Intl phone: 44 1382 385819
FAX (01382) 388072
email: [hidden email]

Lab Page: http://www.lifesci.dundee.ac.uk/people/jason-swedlow
Open Microscopy Environment: http://openmicroscopy.org
**************************

När du skickar e-post till Karolinska Institutet (KI) innebär detta att KI kommer att behandla dina personuppgifter. Här finns information om hur KI behandlar personuppgifter<https://ki.se/medarbetare/integritetsskyddspolicy>.

Sending email to Karolinska Institutet (KI) will result in KI processing your personal data. You can read more about KI’s processing of personal data here<https://ki.se/en/staff/data-protection-policy>.