Posted by
mcammer on
URL: http://confocal-microscopy-list.275.s1.nabble.com/data-storage-requirements-for-NIH-records-original-videos-or-just-tracks-tp7579647p7579654.html
*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy*****
Dear Tim et al.,
Based on meetings I was in during the spring & summer, my employer wants everything saved and IT is moving in the direction of facilitating this. (Right now it is reasonable and practical for us to buy USB HD drives and store them off site; I live 20 miles north of work, so my house is the repository for now.) Based on a quick Google search this morning and the paragraph quoted below, the question isn't only what the government mandates (which may be contradictory depending where you look), but what your university requires. Google turns up a number of university specific guidelines. So, in your case, does UNC have a policy?
Take a look at this vague statement on the Office of Research Integrity web site:
http://ori.dhhs.gov/education/products/RCRintro/c06/3c6.html"Period of retention. Data should be retained for a reasonable period of time to allow other researchers to check results or to use the data for other purposes. There is, however, no common definition of a reasonable period of time. NIH generally requires that data be retained for 3 years following the submission of the final financial report. Some government programs require retention for up to 7 years. A few universities have adopted data-retention policies that set specific time periods in the same range, that is, between 3 and 7 years. Aside from these specific guidelines, however, there is no comprehensive rule for data retention or, when called for, data destruction."
Regards,
Michael
-----Original Message-----
From: Confocal Microscopy List [mailto:
[hidden email]] On Behalf Of Andreas Bruckbauer
Sent: Wednesday, February 06, 2013 3:49 AM
To:
[hidden email]
Subject: Re: data storage requirements for NIH records - original videos or just tracks?
*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy*****
Dear all,
i think this is an important question and also a problem with other microcopy techniques like super-resolution localisation microscopty or SPIM time laps recodings. One aspect is that one might want to reanalise the videos at a later time with an improved tracking algorithm and would need the raw data for this. Lossless compression could help as well as cropping out the important part of the image. But when the amount of data created is just prohibitive one might argue that it should be easer to repeat the experiment than saving all the data. This would mean making sure there will be access to the samples and keeping the microscopes as well as knowledge how to use them...
best wishes
Andreas
-----Original Message-----
From: Dmitry Sokolov <
[hidden email]>
To: CONFOCALMICROSCOPY <
[hidden email]>
Sent: Wed, 6 Feb 2013 8:22
Subject: Re: data storage requirements for NIH records - original videos or just tracks?
*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy*****
Hi Tim,
good question! I believe what you are asking relates to the data sampling.
The scientific method is about the reproducibility of the experiments under the conditions given.
The Technology of Research deals with the sustainability of human
activities:
http://confocal-manawatu.pbworks.com/w/page/48682494/Goals%20of%20Technology%20of%20ResearchIn your case we should probably talk about the sustainability of your research.
If the high data sampling rate makes your research unsustainable, it will not be practical to confirm your results too.
However, if your secondary data (the tracks) are reproducible, you are safe.
Geological expedition is probably a good analogy with your experiment. I believe that its description is satisfactory when based on the GPS data, physical samples and photos from the sampling sites. High definition satellite real-time movies would be probably useful but still not required. The volume and character of data must be adequate to the objectives of a problem. Too much of data in your raw images form the "noise" that is meant to be "filtered" by your particle tracking algorithms. This is the fundamental problem of scientific instrumentation as the human/nature interface.
I hope you find it useful.
Other opinions would be highly appreciated.
Published in MIAWiki:
http://confocal-manawatu.pbworks.com/w/page/63370062/Scientific%20Instrumentation%20as%20Human%20-%20Nature%20InterfaceCheers,
Dmitry
*Advanced Knowledge Management*
for *MICROSCOPY *and *Image Analysis *
------------------------------------------------------------------------
*Dmitry Sokolov*, Ph.D.
Mob: *+64 21 063 5382***
[hidden email] <mailto:
[hidden email]>
06.02.2013 8:32, O'Brien III, E. Timothy ?????:
> *****
> To join, leave or search the confocal microscopy listserv, go to:
>
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy> *****
>
> Dear Microscopists-
>
> Our group has begun using a parallel microscope system to study the
> movement
of fluorescent beads on cells, or in biofilms, mucus, other biological fluids.
We then track the bead movements and generate MSD (mean squared displacement) curves for each bead. Each 1 minute video at 60 FPS takes up about a gigabyte of data storage. Meanwhile the tracks (position/ time) might take several kB for each bead. We can take 12 videos simultaneously, so potentially we are generating 12 gB/minute, a terabyte every hour and 25 minutes!
>
> We believe that taking an image at the beginning of tracking, and
> keeping the
tracking records would be sufficient for us to troubleshoot our data, since we can't possibly store the original videos. This would let us know where the beads were at the beginning of the video (on the nucleus? On the glass?) Signatures of "lost beads" or "stuck beads" are easily identified in control experiments.
>
> We are also considering other intermediate data reduction-potentially
> saving
parts of the videos throughout the timecourse. But this is going to be difficult to implement, and keep track of. Moreover, the reduction is not nearly as high as taking one frame and keeping the tracking results.
>
> What is the community's understanding of the requirements for storing
"original" data? Do we need to keep full videos and spend all our budget on hard drives, or will just the position/time data and an index frame be enough?
>
> What other solutions does your group use?
>
> Thanks very much!
>
> Tim O'Brien
> Computer Integrated Systems for Microscopy and Manipulation UNC Chapel
> Hill, North Carolina