I.T. questions
Locked
|
Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful!
Thanks!! Mike |
|
|
I have users bring a USB thumb-drive, USB hard-disk or blank CDs ;)
Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! > > Thanks!! > Mike > > > |
|
|
In reply to this post by Mike Tighe
This is a good idea, but be wary. We had a core's system blow up due to an infected external hard drive. |
|
|
In reply to this post by Peter Carroll
We recently had a problem to move a single 4GB file by using memory sticks or external harddisks, the file system on these devices did not allow such large files. The systemt is not connected to the network because it does not run anti virus software. We then moved the file to a laptop computer (MAC) using a network cable.
best wishes Andreas From: Peter Carroll <[hidden email]> To: [hidden email] Sent: Thu, 9 Jul 2009 21:53 Subject: Re: I.T. questions
I have users bring a USB thumb-drive, USB hard-disk or blank CDs ;)
Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! > > Thanks!! > Mike > > > Click here to get the very best of AOL, including news, sport, gossip, lifestyles updates and email. |
|
|
This is a problem with the FAT32 file system; it only allows 2GB file sizes. We've run into this problem as well.
In my core, we allow users to store data on local hard drives for up to 30 days. This is enforced by a script that runs nightly and deletes any data older than that. Until recently we also provided a 3 TB file server for users to temporarily store data on; this had a 120 day limit for storing data. The 120 day limit was necessary to keep this system from filling up; the data rate from our core of 6 microscopes (including 2 confocals and a time lapse system) was 6-9 TB / year. That server recently died and we're in the process of replacing it with a larger file server. Kurt ________________________________________ From: Confocal Microscopy List [[hidden email]] On Behalf Of Andreas Bruckbauer [[hidden email]] Sent: Thursday, July 09, 2009 2:12 PM To: [hidden email] Subject: Re: I.T. questions We recently had a problem to move a single 4GB file by using memory sticks or external harddisks, the file system on these devices did not allow such large files. The systemt is not connected to the network because it does not run anti virus software. We then moved the file to a laptop computer (MAC) using a network cable. best wishes Andreas -----Original Message----- From: Peter Carroll <[hidden email]> To: [hidden email] Sent: Thu, 9 Jul 2009 21:53 Subject: Re: I.T. questions I have users bring a USB thumb-drive, USB hard-disk or blank CDs ;) Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! > > Thanks!! > Mike > > > ________________________________ Click here<http://www.aol.co.uk/?ncid=acquktaglinehp01> to get the very best of AOL, including news, sport, gossip, lifestyles updates and email. |
|
|
In reply to this post by Mike Tighe
Dear Mike,
We archive everything that is collected on our systems. Our department requires we keep this for 7 years. In the past I have used REO disks, and MO disks. The disks were charged to the PI. I currently use 2 Terabyte Lacie disks which are now really cheap so we do this at no charge. We build the cost into our microscope charge fee. The data on REO disks was burned to CD when that technology was discontinued. We have all data going back to 1994. You would be surprised how many times a faculty member comes and asks if we have something that was done 5 to 10 years ago and they are astonished that I can hand it to them. Users are required to make the archive copy on the disk we provide and to remove their data from scope computers on a monthly basis. I have about 120 users and the system works pretty well. Joe Goodhouse Confocal Core Lab Manager Dept. of Molecular Biology Princeton University 609-258-5432 Visit us at http://www.molbio1.princeton.edu/facility/confocal/ -----Original Message----- From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Mike Tighe Sent: Thursday, July 09, 2009 4:52 PM To: [hidden email] Subject: I.T. questions Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! Thanks!! Mike |
|
|
In reply to this post by Peter Carroll
Yes - be wary of USB devices - they can have viruses or worms.
I would recommend that you talk to the company that supplies you with the image acquisition software to find a compatible antivirus software. Many companies (the three that I have dealt with) initially say that they do not want antivirus software on the system. We all know that viruses are inevitable reality on PCs these days. The more I talk to companies, the more the more they tell you about what they have tested with their software and they can make recommendations for antivirus software. I restrict removal of files from the acquisition computer to a new DVD/CD or transferring to institutional server space. If the files are too big for a DVD, obviously you will need another solution. If computer is not connected to the network for access to a server archive, then you may be left with external hard drives. I would recommend that you require people to plug their external hard drive into another computer with antivirus software running and scan it BEFORE they plug it into the acquisition computer. After a certain amount of time, the files are removed from the acquisition computer. And similarly, users have to remove their files from the facility server space and be responsible for their own backup. I am a new facility and have not filled up my allotted institutional server space, but anticipate enforcing the timely removal rules soon. - Lisa --------------------------------------- Lisa Cameron, Ph.D. Director of Confocal and Light Microscopy Dana Farber Cancer Institute 44 Binney St.; JF 215 Boston, MA 02115 Office phone: 617-582-8824 Fax: 617-582-8750 Office location: Jimmy Fund Bldg. 220B -----Original Message----- From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Peter Carroll Sent: Thursday, July 09, 2009 4:54 PM To: [hidden email] Subject: Re: [CONFOCALMICROSCOPY] I.T. questions I have users bring a USB thumb-drive, USB hard-disk or blank CDs ;) Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! > > Thanks!! > Mike > > > The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. |
|
|
In reply to this post by Mike Tighe
We have a partition separate from the system partition where users are
expected to direct their images; that way they can't jam up the OS. On that partition we have directories named Monday, Tuesday, Wednesday, etc. Data goes into a folder named with a clear form of the user_name under the current day folder. They are allowed to leave data for 7 days and then it gets erased (regardless of disc space - so as to encourage action on things....) Users typically bring a flash drive these days, but they have the confidence that the data is still sitting there if something goes wrong. I suggest that they put it on their computer (from the flashdrive) and inspect the images as soon as possible to make sure all is OK. I believe in old-school archiving protocols, but I don't think many people follow that advice. We also have full internet access to any server they can connect to; Umass has a "Udrive" system that gives each account a fair bit of space, and we have another computer in the lab with open access so they can put it there just to cover all bases. I think the "7-day" system is a good compromise and doesn't leave me the responsibility of data security and full disks. Dale Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! > > Thanks!! > Mike |
|
|
In reply to this post by Andreas Bruckbauer
To avoid the perpetual software upgrades, etc., that seem to come daily, even hourly, our IT people put the instrument computers into a separate network workgroup with different rules. We have some older image capture software that doesn’t run so well with the latest service pack, so we can select which bits to upgrade. One instrument cannot be connected to the network – the internal data transfer from imaging chip to computer is easily mucked up by any ideas the computer might have of pinging the network, so all automatic functions had to be disabled, it’s the only one we have to download via memory stick. It’s a bit of a problem because single data files can be many GB, so we’ve had to use a portable hard drive. Some people generate many GB of data in a session, others much less, it varies quite a bit. I advise people to wait until they are sure they have their data in at least two other places apart from the instrument computer before deleting. Most people are pretty good about deleting their old data, and we’ve rarely had problems with hard drives filling up. Years ago, I’d have to worry about this, but not now – it’s much easier and cheaper (considering the cost of your salary time doing something about it) to just have big hard drives.... cheers, Rosemray Rosemary White CSIRO Plant Industry GPO Box 1600 Canberra, ACT 2601 Australia ph 61 2 6246 5475 fx 61 2 6246 5334 On 10/07/09 7:12 AM, "Andreas Bruckbauer" <bruckbaua@...> wrote: We recently had a problem to move a single 4GB file by using memory sticks or external harddisks, the file system on these devices did not allow such large files. The systemt is not connected to the network because it does not run anti virus software. We then moved the file to a laptop computer (MAC) using a network cable. |
|
|
In reply to this post by Kurt Thorn
Hi Kurt- Is the 30 day erase script something that can be shared? I don't know how to do it myself and that sounds like a great way to solve an old problem. Dave
On Jul 9, 2009, at 5:26 PM, Thorn, Kurt wrote:
Dr. David Knecht Department of Molecular and Cell Biology Co-head Flow Cytometry and Confocal Microscopy Facility U-3125 91 N. Eagleville Rd. University of Connecticut Storrs, CT 06269 860-486-2200 860-486-4331 (fax) |
|
|
In reply to this post by Goodhouse, Joseph G.
Goodhouse, Joseph G. wrote:
> Dear Mike, > We archive everything that is collected on our systems. > Our department requires we keep this for 7 years. In the past I have > used REO disks, and MO disks. The disks were charged to the PI. I > currently use 2 Terabyte Lacie disks which are now really cheap so we do > 3 out of 4 lacie hard disks have been returned from our lab after dying in various ways - it seems more people have had trouble. we have had better experience with seagate. one of our internal hard disks have also blown, western digital. I recommend buying pairs of hard drives (same size, need not/should not be same manfacturer). backup is very simple using a mirroring, and file size/count does not affect it. DVD is a pain; none of our recordings fit, transfer takes time and they age. I am pushing here to transfer data directly to our main server; very hard to motivate local storage, and it usually doesn't get cleaned up enough. our overnight timelapse recordings are sent while recording but that only works due to our custom recording software; if the network goes down then most programs would just die (so for larger files, store locally first, then move directly). get a gigabit network if you don't have one. if you worry about viruses, you can always install linux on the microscope computer. if your software can't handle that, time to look for a better vendor. or have a look at micro-manager. /Johan > this at no charge. We build the cost into our microscope charge fee. > The data on REO disks was burned to CD when that technology was > discontinued. We have all data going back to 1994. You would be > surprised how many times a faculty member comes and asks if we have > something that was done 5 to 10 years ago and they are astonished that I > can hand it to them. > Users are required to make the archive copy on the disk we > provide and to remove their data from scope computers on a monthly > basis. I have about 120 users and the system works pretty well. > > > Joe Goodhouse > Confocal Core Lab Manager > Dept. of Molecular Biology > Princeton University > 609-258-5432 > > Visit us at http://www.molbio1.princeton.edu/facility/confocal/ > > > -----Original Message----- > From: Confocal Microscopy List [mailto:[hidden email]] > On Behalf Of Mike Tighe > Sent: Thursday, July 09, 2009 4:52 PM > To: [hidden email] > Subject: I.T. questions > > Would anyone be willing to share some of your Ideas on managing data > generated on single and multi photon instruments? Do you allow users to > move data directly to institute server? Has anyone tried to estimate the > amount of data per person or per hour of usage that is generated. How > often do you remove data from instrument computer? Any advice would be > very useful! > > Thanks!! > Mike > -- -- ------------------------------------------------ Johan Henriksson MSc Engineering PhD student, Karolinska Institutet http://mahogny.areta.org http://www.endrov.net |
|
|
In reply to this post by Gary Laevsky-2
|
|
|
In reply to this post by Mike Tighe
Our preferred method is secure FTP to a server. 2 servers share a
gibabit ethernet switch with most of our imaging workstations. Low likelihood of viruses unless both the server is compromised and you are using a vulnerable FTP client. A new DVD also poses little risk, but they are slow. We do allow USB flash drives and portable hard drives, but they are vulnerable to worms and I'm weighing the options for banning them. Files are kept for 2 months on imaging system hard drives to give users a chance to check their files before we delete them The highest practical rate of collection for our current confocal users seem to be about 1 to 1.5 gigabyte/hour, taking into account time to set up the imaging parameters, averaging, sequential channel collection, etc. Widefield imaging systems can generate higher data rates, if the specimen can survive. Confocal manufacturers need to pay more attention to the realities in the customer's world and make their software sufficiently robust and well-behaved to accomodate anti-virus software. Kurt, I'd also like to see your script for deleting the files, if you can share it. regards, Glen Glen MacDonald Core for Communication Research Virginia Merrill Bloedel Hearing Research Center Box 357923 University of Washington Seattle, WA 98195-7923 USA (206) 616-4156 [hidden email] ****************************************************************************** The box said "Requires WindowsXP or better", so I bought a Macintosh. ****************************************************************************** On Jul 9, 2009, at 1:51 PM, Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data > generated on single and multi photon instruments? Do you allow users > to move data directly to institute server? Has anyone tried to > estimate the amount of data per person or per hour of usage that is > generated. How often do you remove data from instrument computer? > Any advice would be very useful! > > Thanks!! > Mike |
 
|
Boswell, Carl A - (cboswell) |
Re: I.T. questions
|
|
In reply to this post by Christian-103
We've been using a RAID for the last year or so,
with space allotted to each lab. Periodically all RAID data is backed up
to tape. Now that users are used to moving files to their space, it
has become pretty routine for them.
My experience has been that all users will never
remove all their files, so I don't bother expecting it. Therefore, as a
computer drive fills up, I send out word to everyone that they need to make
sure all files are archived by the next week, then I go in and flush the
system clean. It's accepted that fair warning is, by
definition, fair, and if someone neglects to get their files to a safe place in
time, they get to do the experiment over again. So far, so
good.
C
Carl A. Boswell, Ph.D.
Molecular and Cellular Biology University of Arizona 520-954-7053 FAX 520-621-3709
|
|
|
In reply to this post by Glen MacDonald-2
Glen MacDonald wrote:
> Our preferred method is secure FTP to a server. 2 servers share a > gibabit ethernet switch with most of our imaging workstations. Low > likelihood of viruses I would also like to point out that SFTP these days can be mounted as a network file system on at least linux and mac. can be easier than fiddling with an SFTP-client. /Johan -- -- ------------------------------------------------ Johan Henriksson MSc Engineering PhD student, Karolinska Institutet http://mahogny.areta.org http://www.endrov.net |
|
|
In reply to this post by Mike Tighe
We have our own domain and IT staff, so users by default get access to
our file server after login. Our confocal users save directly to the file server, right now we have 2 Leica Sp2s which are not very sophisticated from a networking perspective. With all other systems, users move their saved data to the file server at the end of their session. We have several workstations for post processing which are setup so users profiles are redirected to file server by default no actual local profiles on the workstations. All systems have a local partition just for editing large files. Users, are allowed to bring their own disk (USB or Fire Wire). We will even allow direct network connections to the users servers after a security review by IT staff. We allow users to keep files on the servers until they leave the university. We back up all data to two huge tape libraries. Data that users delete is marked for permanent removal from backups after 90days. We only back up user data and very specific parts of our system configurations and whatever modifications we had to make to get the system to work in our network. We don't add extra cost to usage for memory usage at this time. We charge hourly for all people logged into a microscope and we charge monthly for all people logged into our workstations. Until recently, 6TB of storage for users was adequate for about 1,000 users. This hovered between 5TB to 5.8TB over the past year so we may add more storage next year. We have another storage area that is the same size for special users with unique storage needs mainly from our micro, & nano CT machines, these people are assessed fees based on their projects. Data on local machines is allowed to stay until the user leaves the university. We do ask them to clean up their local data as courtesy but don't enforce it. We tend to just add more storage to the local machine if it is needed. As for data removal, we created scripts that search the university servers for status of a user, if the user is no longer with the university their server profile and data are removed, this script is run twice a year and users have 90days to contact us and get their data after they leave the university. Before running the script we will make an effort to contact the users on the list the script generates. If a research group builds a "group profile" with us, their data stays until the PI of the group leaves the university, so if their grad students and postdocs leave their data will remain as long as it is in the group profile, their specific personal profiles are still removed from elsewhere on our network. Does this help? On Jul 9, 2009, at 3:51 PM, Mike Tighe wrote: > Would anyone be willing to share some of your Ideas on managing data > generated on single and multi photon instruments? Do you allow users > to move data directly to institute server? Has anyone tried to > estimate the amount of data per person or per hour of usage that is > generated. How often do you remove data from instrument computer? > Any advice would be very useful! > > Thanks!! > Mike Jon Ekman [hidden email] Imaging Technology Group Beckman Institute for Advanced Science and Technology University of Illinois at Urbana-Champaign 405 N. Mathews Avenue Urbana, IL 61801 USA Tel: 217-244-6292 Fax: 217-244-6219 http://www.itg.uiuc.edu/ |
|
|
In reply to this post by mahogny
> if you worry about viruses, you can always install linux on the Your application does not need to run under linux. Here only the file-server runs linux. The Confocal PC is only connected to it and is thus protected from network viruses. Users retrive their data from the linux computer which is accessible by network. I was not involved in implementing the connections but I can forward questions if somebody needs more details.
> > > 3 out of 4 lacie hard disks have been returned from our lab after > dyingin various ways - it seems more people have had trouble. we > have had > better experience with seagate. one of our internal hard disks > have also > blown, western digital. > > I recommend buying pairs of hard drives (same size, need > not/should not > be same manfacturer). backup is very simple using a mirroring, and > filesize/count does not affect it. DVD is a pain; none of our > recordingsfit, transfer takes time and they age. > > I am pushing here to transfer data directly to our main server; very > hard to motivate local storage, and it usually doesn't get cleaned up > enough. our overnight timelapse recordings are sent while > recording but > that only works due to our custom recording software; if the network > storelocally first, then move directly). get a gigabit network if > you don't > have one. > > if you worry about viruses, you can always install linux on the > microscope computer. if your software can't handle that, time to look > for a better vendor. or have a look at micro-manager. > > /Johan > > this at no charge. We build the cost into our microscope charge > fee.> The data on REO disks was burned to CD when that technology was > > discontinued. We have all data going back to 1994. You would be > > surprised how many times a faculty member comes and asks if we have > > something that was done 5 to 10 years ago and they are > astonished that I > > can hand it to them. > > Users are required to make the archive copy on the disk we > > basis. I have about 120 users and the system works pretty well. > > > > > > Joe Goodhouse > > Confocal Core Lab Manager > > Dept. of Molecular Biology > > Princeton University > > 609-258-5432 > > > > Visit us at http://www.molbio1.princeton.edu/facility/confocal/ > > > > > > > -----Original Message----- > > From: Confocal Microscopy List > [mailto:[hidden email]]> On Behalf Of Mike Tighe > > Sent: Thursday, July 09, 2009 4:52 PM > > To: [hidden email] > > Subject: I.T. questions > > > > Would anyone be willing to share some of your Ideas on managing data > > generated on single and multi photon instruments? Do you allow > users to > estimate the > > amount of data per person or per hour of usage that is > generated. How > > often do you remove data from instrument computer? Any advice > would be > > very useful! > > > > Thanks!! > > Mike > > > > > -- > -- > ------------------------------------------------ > Johan Henriksson > MSc Engineering > PhD student, Karolinska Institutet > http://mahogny.areta.org http://www.endrov.net > |
|
|
In reply to this post by Knecht, David
For everyone who wanted the file deleter code, I put it up for download at http://nic.ucsf.edu/File deleter.zip
There are two versions in there, you want oldFileDelete02. Run.bat shows you how to run it: basically, you run oldFileDelete02 <directory> <age_threshold_in_days> <0_forNoDelete,1_forDelete> <0=list_deleted,1=list_all> Be careful and don't accidentally delete anything you don't want to. Kurt ________________________________________ From: Confocal Microscopy List [[hidden email]] On Behalf Of David Knecht [[hidden email]] Sent: Thursday, July 09, 2009 2:42 PM To: [hidden email] Subject: Re: I.T. questions Hi Kurt- Is the 30 day erase script something that can be shared? I don't know how to do it myself and that sounds like a great way to solve an old problem. Dave On Jul 9, 2009, at 5:26 PM, Thorn, Kurt wrote: This is a problem with the FAT32 file system; it only allows 2GB file sizes. We've run into this problem as well. In my core, we allow users to store data on local hard drives for up to 30 days. This is enforced by a script that runs nightly and deletes any data older than that. Until recently we also provided a 3 TB file server for users to temporarily store data on; this had a 120 day limit for storing data. The 120 day limit was necessary to keep this system from filling up; the data rate from our core of 6 microscopes (including 2 confocals and a time lapse system) was 6-9 TB / year. That server recently died and we're in the process of replacing it with a larger file server. Kurt ________________________________________ From: Confocal Microscopy List [[hidden email]<mailto:[hidden email]>] On Behalf Of Andreas Bruckbauer [[hidden email]<mailto:[hidden email]>] Sent: Thursday, July 09, 2009 2:12 PM To: [hidden email]<mailto:[hidden email]> Subject: Re: I.T. questions We recently had a problem to move a single 4GB file by using memory sticks or external harddisks, the file system on these devices did not allow such large files. The systemt is not connected to the network because it does not run anti virus software. We then moved the file to a laptop computer (MAC) using a network cable. best wishes Andreas -----Original Message----- From: Peter Carroll <[hidden email]<mailto:[hidden email]>> To: [hidden email]<mailto:[hidden email]> Sent: Thu, 9 Jul 2009 21:53 Subject: Re: I.T. questions I have users bring a USB thumb-drive, USB hard-disk or blank CDs ;) Mike Tighe wrote: Would anyone be willing to share some of your Ideas on managing data generated on single and multi photon instruments? Do you allow users to move data directly to institute server? Has anyone tried to estimate the amount of data per person or per hour of usage that is generated. How often do you remove data from instrument computer? Any advice would be very useful! Thanks!! Mike ________________________________ Click here<http://www.aol.co.uk/?ncid=acquktaglinehp01> to get the very best of AOL, including news, sport, gossip, lifestyles updates and email. Dr. David Knecht Department of Molecular and Cell Biology Co-head Flow Cytometry and Confocal Microscopy Facility U-3125 91 N. Eagleville Rd. University of Connecticut Storrs, CT 06269 860-486-2200 860-486-4331 (fax) |
«
Return to Confocal Microscopy List
|
1 view|%1 views
| Free forum by Nabble | Edit this page |