Data transfer and storage

[simon walker (BI)]

Search the CONFOCAL archive at
http://listserv.acsu.buffalo.edu/cgi-bin/wa?S1=confocal

I'm looking for advice on how to set up a fast local network to transfer
data from pcs used to operate microscopes (acquisition pcs) to pcs used
for analysis (analysis pcs).
Currently we have 5 acquisition pcs connected on a gigabit lan to an
external hard disk (a Thecus device which also acts as a portal to our
main network) and a single analysis pc connected via the gigabit lan to
the same external hard disk.  The problem is that saving data to the
external hard disk from acquisition pcs takes longer than I'd like, and
analysing large data sets (>300 Mb) directly off the external hard disk
(rather than transferring it again onto the analysis pc hard drive) takes
forever.
I'm interested in how other facilities have addressed this problem.  Is it
possible to create a setup where data can be stored and accessed
externally at a speed comparable to internal storage?

Thanks in advance for any helpful suggestions.
Simon
Babraham Institute,
Cambridge.

[David Basiji]

Search the CONFOCAL archive at
http://listserv.acsu.buffalo.edu/cgi-bin/wa?S1=confocal

Simon,

This is very similar to the network architecture we have
here at Amnis and we have no problems with latency during
data upload from our instruments or direct analysis of
data residing on the shared drive.

I'd check a few things:
1. Our shared drive is an Apple Xserve RAID that has
very high transfer bandwidth. I don't know the bandwidth
of the Thecus unit, but it may be a bottleneck.

2. Gigabit demands either Cat6 wiring or shorter runs of
Cat5. If your wiring isn't up to snuff, your gigabit
switches will operate at 10/100 Mbs.

3. Even with good wiring, your gigabit switches may not
be rated to maintain full bandwidth when multiple computers
are transferring data at the same time. Check their specs
to see.

4. It's frustratingly common for Windows PCs to have
incorrect network settings that downgrade their gigbit
cards to operate at 10/100 Mbs.  

I hope this helps.

David

David Basiji, Ph.D.
President and CEO, Amnis Corporation
2505 Third Ave., Suite 210
Seattle, WA 98121
[phone] direct
[phone] mobile
[phone] fax
 
This email and any attachment contain information which is intended for
the addressees only. If you have received this email in error, please
notify the sender.
 

> -----Original Message-----
> From: Confocal Microscopy List
> [mailto:[hidden email]] On Behalf Of Simon Walker
> Sent: Thursday, September 06, 2007 2:13 AM
> To: [hidden email]
> Subject: Data transfer and storage
>
>
> Search the CONFOCAL archive at
> http://listserv.acsu.buffalo.edu/cgi-bin/wa?S1=confocal
>
> I'm looking for advice on how to set up a fast local network
> to transfer
> data from pcs used to operate microscopes (acquisition pcs)
> to pcs used
> for analysis (analysis pcs).
> Currently we have 5 acquisition pcs connected on a gigabit lan to an
> external hard disk (a Thecus device which also acts as a
> portal to our
> main network) and a single analysis pc connected via the
> gigabit lan to
> the same external hard disk.  The problem is that saving data to the
> external hard disk from acquisition pcs takes longer than I'd
> like, and
> analysing large data sets (>300 Mb) directly off the external
> hard disk
> (rather than transferring it again onto the analysis pc hard
> drive) takes
> forever.
> I'm interested in how other facilities have addressed this
> problem.  Is it
> possible to create a setup where data can be stored and accessed
> externally at a speed comparable to internal storage?
>
> Thanks in advance for any helpful suggestions.
> Simon
> Babraham Institute,
> Cambridge.
>

[Jason Swedlow]

Search the CONFOCAL archive at http://listserv.acsu.buffalo.edu/cgi-bin/wa?S1=confocal Hi Simon,

In my experience, this is a case where there are many solutions, and even more opinions.  Seems that everyone and their brother is smarter than everyone else.

The thing to do is to use some measurements-- exactly how long does it take to copy your 300 MB file from the server disk to your local disk.  Once you have that, then you can compare that number with what you expect-- based on the network, the disk, etc.  It's probably important to also know what happens when you get two people accessing the disk at the same time.  Do you have anyone there at the Babraham who is reasonably savvy with this tech?  Someone who knows their way around a network can be very helpful.

The short answer-- yes, you should be able to exceed to performance of your local disk, but you do have to pay for it.  This is what enterprise level SAN systems can provide (for lots of users). 

Cheers,

Jason

On 9/6/07, David Basiji <[hidden email]> wrote:

Search the CONFOCAL archive at
http://listserv.acsu.buffalo.edu/cgi-bin/wa?S1=confocal

Simon,

This is very similar to the network architecture we have
here at Amnis and we have no problems with latency during
data upload from our instruments or direct analysis of
data residing on the shared drive.

I'd check a few things:
1. Our shared drive is an Apple Xserve RAID that has
very high transfer bandwidth. I don't know the bandwidth
of the Thecus unit, but it may be a bottleneck.

2. Gigabit demands either Cat6 wiring or shorter runs of
Cat5. If your wiring isn't up to snuff, your gigabit
switches will operate at 10/100 Mbs.

3. Even with good wiring, your gigabit switches may not
be rated to maintain full bandwidth when multiple computers
are transferring data at the same time. Check their specs
to see.

4. It's frustratingly common for Windows PCs to have
incorrect network settings that downgrade their gigbit
cards to operate at 10/100 Mbs.

I hope this helps.

David

David Basiji, Ph.D.
President and CEO, Amnis Corporation
2505 Third Ave., Suite 210
Seattle, WA 98121
[phone] direct
[phone] mobile
[phone] fax

This email and any attachment contain information which is intended for
the addressees only. If you have received this email in error, please
notify the sender.

> -----Original Message-----
> From: Confocal Microscopy List
> [mailto:[hidden email]] On Behalf Of Simon Walker
> Sent: Thursday, September 06, 2007 2:13 AM
> To: [hidden email]
> Subject: Data transfer and storage
>
>
> Search the CONFOCAL archive at
> http://listserv.acsu.buffalo.edu/cgi-bin/wa?S1=confocal
>
> I'm looking for advice on how to set up a fast local network
> to transfer
> data from pcs used to operate microscopes (acquisition pcs)
> to pcs used
> for analysis (analysis pcs).
> Currently we have 5 acquisition pcs connected on a gigabit lan to an
> external hard disk (a Thecus device which also acts as a
> portal to our
> main network) and a single analysis pc connected via the
> gigabit lan to
> the same external hard disk.  The problem is that saving data to the
> external hard disk from acquisition pcs takes longer than I'd
> like, and
> analysing large data sets (>300 Mb) directly off the external
> hard disk
> (rather than transferring it again onto the analysis pc hard
> drive) takes
> forever.
> I'm interested in how other facilities have addressed this
> problem.  Is it
> possible to create a setup where data can be stored and accessed
> externally at a speed comparable to internal storage?
>
> Thanks in advance for any helpful suggestions.
> Simon
> Babraham Institute,
> Cambridge.
>