Confocal Microscopy List

data backup solutions

Classic

List

Threaded

10 messages Options

jcv2@uw.edu

data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Confocalists,

Do any of you have an approach for data backup on the scale of a few tens of TB that you like?

We have ~20 TB at present and I anticipate getting maybe another 15 TB per year over the next few years. We purchased a NAS array a few years ago which worked fine but is now laughably small and much of our raw data is not backed up at present.

Cloud storage is appealing since it would enable our backup to grow as we grow, but from a quick search, it seems to be somewhat expensive. That is, over even one year, a DIY approach would probably be somewhat cheaper, let alone over a few years.

We keep our data relatively centralized on one server that can be accessed simultaneously by many users. We may get a second one sometime.

Best regards,
Josh

Craig Brideau

Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Our lab has a couple QNAP servers. They can have 8+ drive bays each, so if
you fill it with 8TB drives you get 64TB of storage per chassis, and you
can add an extra 8-bay expansion to any chassis. I find 8TB drives are
about at the sweet spot for cost/capacity right now.

Craig

On Tue, Aug 29, 2017 at 2:56 PM, Joshua Vaughan <[hidden email]> wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Dear Confocalists,
>
> Do any of you have an approach for data backup on the scale of a few tens
> of TB that you like?
>
> We have ~20 TB at present and I anticipate getting maybe another 15 TB per
> year over the next few years. We purchased a NAS array a few years ago
> which worked fine but is now laughably small and much of our raw data is
> not backed up at present.
>
> Cloud storage is appealing since it would enable our backup to grow as we
> grow, but from a quick search, it seems to be somewhat expensive. That is,
> over even one year, a DIY approach would probably be somewhat cheaper, let
> alone over a few years.
>
> We keep our data relatively centralized on one server that can be accessed
> simultaneously by many users. We may get a second one sometime.
>
>
> Best regards,
> Josh
>

Beth Cimini

Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Depending on how often you think you'll need to access the data, if it's
truly just as backup our lab (about 30TB of archival data and another ~30TB
of "active" data)has had pretty good luck just using AWS Glacier for the
archival stuff. It's currently $48/TB/year, so at your size about $1K per
year (perhaps a bit less if you archive with compression), but for our lab
it was worth the peace of mind to not have to worry about disk failures,
updating infrastructure, etc etc.

On Tue, Aug 29, 2017 at 5:34 PM, Craig Brideau <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your posting.
> *****
>
> Our lab has a couple QNAP servers. They can have 8+ drive bays each, so if
> you fill it with 8TB drives you get 64TB of storage per chassis, and you
> can add an extra 8-bay expansion to any chassis. I find 8TB drives are
> about at the sweet spot for cost/capacity right now.
>
> Craig
>
> On Tue, Aug 29, 2017 at 2:56 PM, Joshua Vaughan <[hidden email]> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> > Post images on http://www.imgur.com and include the link in your
> posting.
> > *****
> >
> > Dear Confocalists,
> >
> > Do any of you have an approach for data backup on the scale of a few tens
> > of TB that you like?
> >
> > We have ~20 TB at present and I anticipate getting maybe another 15 TB
> per
> > year over the next few years. We purchased a NAS array a few years ago
> > which worked fine but is now laughably small and much of our raw data is
> > not backed up at present.
> >
> > Cloud storage is appealing since it would enable our backup to grow as we
> > grow, but from a quick search, it seems to be somewhat expensive. That
> is,
> > over even one year, a DIY approach would probably be somewhat cheaper,
> let
> > alone over a few years.
> >
> > We keep our data relatively centralized on one server that can be
> accessed
> > simultaneously by many users. We may get a second one sometime.
> >
> >
> > Best regards,
> > Josh
> >
>

--
Beth Cimini, PhD
Computational Biologist, Imaging Platform
Broad Institute
415 Main St Room 5011
Cambridge, MA 02142

mcammer

Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

What about disaster recovery? Such as when there is a leak from the floor above into your primary server. We had a case here a few years ago where the primary server in the basement was submerged by a storm surge.

Michael Cammer, Research Scientist, DART Microscopy Laboratory
NYU Langone Health, 540 First Avenue, SK2 Microscopy Suite, New York, NY 10016
C: 914-309-3270 [hidden email] http://microscopynotes.com/
https://med.nyu.edu/research/research-resources/scientific-cores-shared-resources/microscopy-laboratory

-----Original Message-----
From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Beth Cimini
Sent: Tuesday, August 29, 2017 5:47 PM
To: [hidden email]
Subject: Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=_mycQZsyYU4MSBcJ3gPYFWBITXlyCAxl1RfVqWBCqX4&e=
Post images on https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=fHQWC2jBywLSoborTh-tWGv9ut1m_Qr4ddBKd8JxSgs&e= and include the link in your posting.
*****

Depending on how often you think you'll need to access the data, if it's truly just as backup our lab (about 30TB of archival data and another ~30TB of "active" data)has had pretty good luck just using AWS Glacier for the archival stuff. It's currently $48/TB/year, so at your size about $1K per year (perhaps a bit less if you archive with compression), but for our lab it was worth the peace of mind to not have to worry about disk failures, updating infrastructure, etc etc.

On Tue, Aug 29, 2017 at 5:34 PM, Craig Brideau <[hidden email]>
wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-
> 2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOB
> Gmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=eP
> FekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=_mycQZsyYU4MSBcJ3gPYFWBITX
> lyCAxl1RfVqWBCqX4&e= Post images on
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=fHQWC2jBywLSoborTh-tWGv9ut1m_Qr4ddBKd8JxSgs&e= and include the link in your posting.
> *****
>
> Our lab has a couple QNAP servers. They can have 8+ drive bays each,
> so if you fill it with 8TB drives you get 64TB of storage per chassis,
> and you can add an extra 8-bay expansion to any chassis. I find 8TB
> drives are about at the sweet spot for cost/capacity right now.
>
> Craig
>
> On Tue, Aug 29, 2017 at 2:56 PM, Joshua Vaughan <[hidden email]> wrote:
>
> > *****
> > To join, leave or search the confocal microscopy listserv, go to:
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cg
> > i-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQIBaQ&c=j5oPpO0eBH1iio48Dtse
> > dbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKku
> > o&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=_mycQZsyYU4MSBcJ3g
> > PYFWBITXlyCAxl1RfVqWBCqX4&e= Post images on
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=
> > DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAy
> > dlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyo
> > wbSbr2E8&s=fHQWC2jBywLSoborTh-tWGv9ut1m_Qr4ddBKd8JxSgs&e= and
> > include the link in your
> posting.
> > *****
> >
> > Dear Confocalists,
> >
> > Do any of you have an approach for data backup on the scale of a few
> > tens of TB that you like?
> >
> > We have ~20 TB at present and I anticipate getting maybe another 15
> > TB
> per
> > year over the next few years. We purchased a NAS array a few years
> > ago which worked fine but is now laughably small and much of our raw
> > data is not backed up at present.
> >
> > Cloud storage is appealing since it would enable our backup to grow
> > as we grow, but from a quick search, it seems to be somewhat
> > expensive. That
> is,
> > over even one year, a DIY approach would probably be somewhat
> > cheaper,
> let
> > alone over a few years.
> >
> > We keep our data relatively centralized on one server that can be
> accessed
> > simultaneously by many users. We may get a second one sometime.
> >
> >
> > Best regards,
> > Josh
> >
>

--
Beth Cimini, PhD
Computational Biologist, Imaging Platform Broad Institute
415 Main St Room 5011
Cambridge, MA 02142

------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================

Johannes Helm

Re: data backup solutions

In reply to this post by jcv2@uw.edu

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Good evening, Joshua,

there is another aspect of "data storage", which you might regard as
"important".

While I am not sure about how the legal situation in your country is,
there are countries where you are forced by law to make sure that any
data relevant for any publication done by your lab are retrievable for a
certain number of years at least.
Given this legal obligation and given the fact that data storage, which
is both, efficient AND safe, is a field of science by itself (sic!), it
might be worth considering to leave this issue to the experts in the
field, in casu your local IT department. They can establish - and
probably you would have to pay for it, but the price is small as
compared to the trouble you can encounter when loosing data - a fast
wire from the local subnet, which your lab computers are a part of, and
a truly safe storage server. When that server approaches its end of use
period, they will also make sure that your data are transferred to new
media, available then. Since it is their job to ensure safe data
storage, the legal responsibility will be theirs, as soon as you handle
the problem over to them. Establishing a system like that will result in
some paper work and you will have to spend more time on the issue than
when doing it all on your own, but it might be worth the effort.

Best wishes,

Johannes

PS: Up here, we have something called NorStore, soon to be replaced with
something new called NIRD, and, while it was quite some work and a
couple of lengthy meetings to get things installed in the proper way,
plus paying for truly fast switches, it works nicely. Physically, the
data are stored "somewhere in the country", and it is not so easy for
any kind of hardware or software thieves to access these data,
manipulate or destroy them, neither is the specific region, where the
data physically are stored known to be exposed to frequent natural
disasters.

On 2017-08-29 22:56, Joshua Vaughan wrote:

> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> Post images on http://www.imgur.com and include the link in your
> posting.
> *****
>
> Dear Confocalists,
>
> Do any of you have an approach for data backup on the scale of a few
> tens of TB that you like?
>
> We have ~20 TB at present and I anticipate getting maybe another 15 TB
> per year over the next few years. We purchased a NAS array a few years
> ago which worked fine but is now laughably small and much of our raw
> data is not backed up at present.
>
> Cloud storage is appealing since it would enable our backup to grow as
> we grow, but from a quick search, it seems to be somewhat expensive.
> That is, over even one year, a DIY approach would probably be somewhat
> cheaper, let alone over a few years.
>
> We keep our data relatively centralized on one server that can be
> accessed simultaneously by many users. We may get a second one
> sometime.
>
>
> Best regards,
> Josh

--
P. Johannes Helm

Voice: (+47) 228 51159 (office)
Fax: (+47) 228 51499 (office)

Christopher Yip

Re: data backup solutions

In reply to this post by mcammer

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

For our lab, we have replicate backups in several different physical locations on campus…

Christopher Yip PhD, P.Eng,

Professor
Institute of Biomaterials and Biomedical Engineering
Department of Chemical Engineering and Applied Chemistry
Department of Biochemistry
The Terrence Donnelly Centre for Cellular and Biomolecular Research
University of Toronto
404 - 160 College St
Toronto, Ontario, CANADA M5S 3E1
(416) 978-7853 (office)
(416) 978-4317 (fax)
[hidden email]<mailto:[hidden email]>
http://bigten.med.utoronto.ca
http://www.thedonnellycentre.utoronto.ca

On Aug 29, 2017, at 6:04 PM, Cammer, Michael <[hidden email]<mailto:[hidden email]>> wrote:

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com<http://www.imgur.com/> and include the link in your posting.
*****

What about disaster recovery? Such as when there is a leak from the floor above into your primary server. We had a case here a few years ago where the primary server in the basement was submerged by a storm surge.

Michael Cammer, Research Scientist, DART Microscopy Laboratory
NYU Langone Health, 540 First Avenue, SK2 Microscopy Suite, New York, NY 10016
C: 914-309-3270 [hidden email]<mailto:[hidden email]> http://microscopynotes.com/
https://med.nyu.edu/research/research-resources/scientific-cores-shared-resources/microscopy-laboratory

-----Original Message-----
From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Beth Cimini
Sent: Tuesday, August 29, 2017 5:47 PM
To: [hidden email]<mailto:[hidden email]>
Subject: Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=_mycQZsyYU4MSBcJ3gPYFWBITXlyCAxl1RfVqWBCqX4&e=
Post images on https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=fHQWC2jBywLSoborTh-tWGv9ut1m_Qr4ddBKd8JxSgs&e= and include the link in your posting.
*****

Depending on how often you think you'll need to access the data, if it's truly just as backup our lab (about 30TB of archival data and another ~30TB of "active" data)has had pretty good luck just using AWS Glacier for the archival stuff. It's currently $48/TB/year, so at your size about $1K per year (perhaps a bit less if you archive with compression), but for our lab it was worth the peace of mind to not have to worry about disk failures, updating infrastructure, etc etc.

On Tue, Aug 29, 2017 at 5:34 PM, Craig Brideau <[hidden email]<mailto:[hidden email]>>
wrote:

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-
2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOB
Gmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=eP
FekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=_mycQZsyYU4MSBcJ3gPYFWBITX
lyCAxl1RfVqWBCqX4&e= Post images on
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=fHQWC2jBywLSoborTh-tWGv9ut1m_Qr4ddBKd8JxSgs&e= and include the link in your posting.
*****

Our lab has a couple QNAP servers. They can have 8+ drive bays each,
so if you fill it with 8TB drives you get 64TB of storage per chassis,
and you can add an extra 8-bay expansion to any chassis. I find 8TB
drives are about at the sweet spot for cost/capacity right now.

Craig

On Tue, Aug 29, 2017 at 2:56 PM, Joshua Vaughan <[hidden email]<mailto:[hidden email]>> wrote:

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cg
i-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQIBaQ&c=j5oPpO0eBH1iio48Dtse
dbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKku
o&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyowbSbr2E8&s=_mycQZsyYU4MSBcJ3g
PYFWBITXlyCAxl1RfVqWBCqX4&e= Post images on
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=
DQIBaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAy
dlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=ePFekrLL_MzGwY765wRCDw3Dom4bAWPfYyo
wbSbr2E8&s=fHQWC2jBywLSoborTh-tWGv9ut1m_Qr4ddBKd8JxSgs&e= and
include the link in your
posting.
*****

Dear Confocalists,

Do any of you have an approach for data backup on the scale of a few
tens of TB that you like?

We have ~20 TB at present and I anticipate getting maybe another 15
TB
per
year over the next few years. We purchased a NAS array a few years
ago which worked fine but is now laughably small and much of our raw
data is not backed up at present.

Cloud storage is appealing since it would enable our backup to grow
as we grow, but from a quick search, it seems to be somewhat
expensive. That
is,
over even one year, a DIY approach would probably be somewhat
cheaper,
let
alone over a few years.

We keep our data relatively centralized on one server that can be
accessed
simultaneously by many users. We may get a second one sometime.

Best regards,
Josh

--
Beth Cimini, PhD
Computational Biologist, Imaging Platform Broad Institute
415 Main St Room 5011
Cambridge, MA 02142

------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================

Christopher Yip

Re: data backup solutions

In reply to this post by jcv2@uw.edu

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

We use FreeNAS (www.freenas.org<http://www.freenas.org>)

Great solution - get a used server (eBay / MrRackables).. and 8TB drives (as was suggested by Craig)… and you’re good to go…

I’ve had great success with this platform… Runs off a USB stick, ZFS so good redundancy in case of hardware failure, great hardware support and really easy to deploy. Has a very nice and complete set of plug-ins. Performance is really quite good. …

Super robust

Chris

Christopher Yip PhD, P.Eng,

Professor
Institute of Biomaterials and Biomedical Engineering
Department of Chemical Engineering and Applied Chemistry
Department of Biochemistry
The Terrence Donnelly Centre for Cellular and Biomolecular Research
University of Toronto
404 - 160 College St
Toronto, Ontario, CANADA M5S 3E1
(416) 978-7853 (office)
(416) 978-4317 (fax)
[hidden email]<mailto:[hidden email]>
http://bigten.med.utoronto.ca
http://www.thedonnellycentre.utoronto.ca

On Aug 29, 2017, at 4:56 PM, Joshua Vaughan <[hidden email]<mailto:[hidden email]>> wrote:

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Confocalists,

Do any of you have an approach for data backup on the scale of a few tens of TB that you like?

We have ~20 TB at present and I anticipate getting maybe another 15 TB per year over the next few years. We purchased a NAS array a few years ago which worked fine but is now laughably small and much of our raw data is not backed up at present.

Cloud storage is appealing since it would enable our backup to grow as we grow, but from a quick search, it seems to be somewhat expensive. That is, over even one year, a DIY approach would probably be somewhat cheaper, let alone over a few years.

We keep our data relatively centralized on one server that can be accessed simultaneously by many users. We may get a second one sometime.

Best regards,
Josh

mcammer

Re: data backup solutions

In reply to this post by Johannes Helm

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

I keep hearing about this (administrators librarians (a.k.a. data scientists), articles in journals, listserv postings, etc) but does anyone on this list know of an author ever having their raw data audited or demanded by a journal, gov't, or other official external entity except in a case of large fraud inquiry? Seems to me just not done. And even not convinced most published software is checked by the reviewers to see if it works.

The only experience I had with the raw data question was approx 2 years ago when a lab here wanted to reproduce a published microscopy method. Having difficulty following the method, to try to clarify we looked at the raw data linked from the publication and found that 1.) the raw data on the server were not those shown as final in the paper, 2.) the corresponding author did not respond to requests for the correct data, and 3.) the journal editor did not want to get involved. We dropped the issue and worked out a method ourselves.

Michael Cammer, Research Scientist, DART Microscopy Laboratory
NYU Langone Health, 540 First Avenue, SK2 Microscopy Suite, New York, NY 10016
C: 914-309-3270 [hidden email] http://microscopynotes.com/
https://med.nyu.edu/research/research-resources/scientific-cores-shared-resources/microscopy-laboratory

-----Original Message-----
From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Johannes Helm
Sent: Tuesday, August 29, 2017 6:26 PM
To: [hidden email]
Subject: Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQICAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=X0TabEZk4DXuQcC2ZoNc59kjoreD0rn976HaYX8SpUs&s=-HZhnwrngJ4ONrvklOYuArALHIqYQrWzrhXgj09xxFc&e=
Post images on https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQICAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=X0TabEZk4DXuQcC2ZoNc59kjoreD0rn976HaYX8SpUs&s=oFXOVG3WUoRWkpFybi8YTFZIr-p7susHgz7Eo3uFnyw&e= and include the link in your posting.
*****

Good evening, Joshua,

there is another aspect of "data storage", which you might regard as "important".

While I am not sure about how the legal situation in your country is, there are countries where you are forced by law to make sure that any data relevant for any publication done by your lab are retrievable for a certain number of years at least.
Given this legal obligation and given the fact that data storage, which is both, efficient AND safe, is a field of science by itself (sic!), it might be worth considering to leave this issue to the experts in the field, in casu your local IT department. They can establish - and probably you would have to pay for it, but the price is small as compared to the trouble you can encounter when loosing data - a fast wire from the local subnet, which your lab computers are a part of, and a truly safe storage server. When that server approaches its end of use period, they will also make sure that your data are transferred to new media, available then. Since it is their job to ensure safe data storage, the legal responsibility will be theirs, as soon as you handle the problem over to them. Establishing a system like that will result in some paper work and you will have to spend more time on the issue than when doing it all on your own, but it might be worth the effort.

Best wishes,

Johannes

PS: Up here, we have something called NorStore, soon to be replaced with something new called NIRD, and, while it was quite some work and a couple of lengthy meetings to get things installed in the proper way, plus paying for truly fast switches, it works nicely. Physically, the data are stored "somewhere in the country", and it is not so easy for any kind of hardware or software thieves to access these data, manipulate or destroy them, neither is the specific region, where the data physically are stored known to be exposed to frequent natural disasters.

On 2017-08-29 22:56, Joshua Vaughan wrote:

--
P. Johannes Helm

Voice: (+47) 228 51159 (office)
Fax: (+47) 228 51499 (office)

------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================

Johannes Helm

Re: data backup solutions

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Dear Michael,

you definitely are right that this, at first glance, might appear odd.

As far as I know, this legally (!) could be important in case of

- patent issues,

- if somebody claims that your publication would be based on "wrong
data" or "wrong assumptions",

- if somebody claims you would have "manipulated" your original data.

I am not trained in law and persons trained in law sometimes appear to
live in "their own world" (and they might claim that scientists are
living in THEIR own world). I personally, just as is your experience,
never had any issues, which would have forced me to re-present my
original data.

Best,
Johannes

On 2017-08-30 19:52, Cammer, Michael wrote:

> I keep hearing about this (administrators librarians (a.k.a. data
> scientists), articles in journals, listserv postings, etc) but does
> anyone on this list know of an author ever having their raw data
> audited or demanded by a journal, gov't, or other official external
> entity except in a case of large fraud inquiry? Seems to me just not
> done. And even not convinced most published software is checked by
> the reviewers to see if it works.
>
> The only experience I had with the raw data question was approx 2
> years ago when a lab here wanted to reproduce a published microscopy
> method. Having difficulty following the method, to try to clarify we
> looked at the raw data linked from the publication and found that 1.)
> the raw data on the server were not those shown as final in the paper,
> 2.) the corresponding author did not respond to requests for the
> correct data, and 3.) the journal editor did not want to get involved.
> We dropped the issue and worked out a method ourselves.
>
> Michael Cammer, Research Scientist, DART Microscopy Laboratory
> NYU Langone Health, 540 First Avenue, SK2 Microscopy Suite, New York,
> NY 10016
> C: 914-309-3270 [hidden email]
> http://microscopynotes.com/
> https://med.nyu.edu/research/research-resources/scientific-cores-shared-resources/microscopy-laboratory
>
>
>
> -----Original Message-----
> From: Confocal Microscopy List
> [mailto:[hidden email]] On Behalf Of Johannes Helm
> Sent: Tuesday, August 29, 2017 6:26 PM
> To: [hidden email]
> Subject: Re: data backup solutions
>
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQICAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=X0TabEZk4DXuQcC2ZoNc59kjoreD0rn976HaYX8SpUs&s=-HZhnwrngJ4ONrvklOYuArALHIqYQrWzrhXgj09xxFc&e=
> Post images on
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQICAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=X0TabEZk4DXuQcC2ZoNc59kjoreD0rn976HaYX8SpUs&s=oFXOVG3WUoRWkpFybi8YTFZIr-p7susHgz7Eo3uFnyw&e=
> and include the link in your posting.
> *****
>
> Good evening, Joshua,
>
> there is another aspect of "data storage", which you might regard as
> "important".
>
> While I am not sure about how the legal situation in your country is,
> there are countries where you are forced by law to make sure that any
> data relevant for any publication done by your lab are retrievable for
> a certain number of years at least.
> Given this legal obligation and given the fact that data storage,
> which is both, efficient AND safe, is a field of science by itself
> (sic!), it might be worth considering to leave this issue to the
> experts in the field, in casu your local IT department. They can
> establish - and probably you would have to pay for it, but the price
> is small as compared to the trouble you can encounter when loosing
> data - a fast wire from the local subnet, which your lab computers are
> a part of, and a truly safe storage server. When that server
> approaches its end of use period, they will also make sure that your
> data are transferred to new media, available then. Since it is their
> job to ensure safe data storage, the legal responsibility will be
> theirs, as soon as you handle the problem over to them. Establishing a
> system like that will result in some paper work and you will have to
> spend more time on the issue than when doing it all on your own, but
> it might be worth the effort.
>
> Best wishes,
>
> Johannes
>
> PS: Up here, we have something called NorStore, soon to be replaced
> with something new called NIRD, and, while it was quite some work and
> a couple of lengthy meetings to get things installed in the proper
> way, plus paying for truly fast switches, it works nicely. Physically,
> the data are stored "somewhere in the country", and it is not so easy
> for any kind of hardware or software thieves to access these data,
> manipulate or destroy them, neither is the specific region, where the
> data physically are stored known to be exposed to frequent natural
> disasters.
>
>
> On 2017-08-29 22:56, Joshua Vaughan wrote:
>> *****
>> To join, leave or search the confocal microscopy listserv, go to:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.umn.edu_cgi-
>> 2Dbin_wa-3FA0-3Dconfocalmicroscopy&d=DQICAg&c=j5oPpO0eBH1iio48DtsedbOB
>> Gmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=X0
>> TabEZk4DXuQcC2ZoNc59kjoreD0rn976HaYX8SpUs&s=-HZhnwrngJ4ONrvklOYuArALHI
>> qYQrWzrhXgj09xxFc&e= Post images on
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imgur.com&d=DQ
>> ICAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=X0TabEZk4DXuQcC2ZoNc59kjoreD0rn976HaYX8SpUs&s=oFXOVG3WUoRWkpFybi8YTFZIr-p7susHgz7Eo3uFnyw&e=
>> and include the link in your posting.
>> *****
>>
>> Dear Confocalists,
>>
>> Do any of you have an approach for data backup on the scale of a few
>> tens of TB that you like?
>>
>> We have ~20 TB at present and I anticipate getting maybe another 15 TB
>> per year over the next few years. We purchased a NAS array a few years
>> ago which worked fine but is now laughably small and much of our raw
>> data is not backed up at present.
>>
>> Cloud storage is appealing since it would enable our backup to grow as
>> we grow, but from a quick search, it seems to be somewhat expensive.
>> That is, over even one year, a DIY approach would probably be somewhat
>> cheaper, let alone over a few years.
>>
>> We keep our data relatively centralized on one server that can be
>> accessed simultaneously by many users. We may get a second one
>> sometime.
>>
>>
>> Best regards,
>> Josh
>
> --
> P. Johannes Helm
>
> Voice: (+47) 228 51159 (office)
> Fax: (+47) 228 51499 (office)
>
> ------------------------------------------------------------
> This email message, including any attachments, is for the sole use of
> the intended recipient(s) and may contain information that is
> proprietary, confidential, and exempt from disclosure under applicable
> law. Any unauthorized review, use, disclosure, or distribution is
> prohibited. If you have received this email in error please notify the
> sender by return email and delete the original message. Please note,
> the recipient should check this email and any attachments for the
> presence of viruses. The organization accepts no liability for any
> damage caused by any virus transmitted by this email.
> =================================

--
P. Johannes Helm

Voice: (+47) 228 51159 (office)
Fax: (+47) 228 51499 (office)

RJ3

Re: data backup solutions

In reply to this post by jcv2@uw.edu

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

Off the shelf consumer NAS solutions are not robust, made with low p RAM, have flaky controllers, and are limited in expandability. Keep the data in your control and DIY by building your own.

ZFS is the most advanced file system in the world, and personally I think the best implementation of it is FreeNAS. It is free and opensource so you'll never get vendor locked.

Main points:
1) ZFS supports RAIDZ3: it can sustain up to 3 drive failures without data loss.
2) It doesn't need a fast processor, but it needs lots of ECC RAM. It also performs it's own error checking to avoid bit rot.
3) Easily expandable by swapping drives with newer ones and resilvering.
4) Tons of plugins available, like NextCloud and SMB services so you can access your data on the internet or on your LAN.

Once setup they last for years with little to no maintenance. I setup my first one in 2011 and it's still running fine. I even built one at home for my own data, at this point I trust nothing else. Ideally you'll build two and keep them in physically different locations and synchronized. The probability of losing data with this setup is close to impossible.