I.T. questions
Locked
|
Hi Mike,
Good question.... it gives me chance to vent some of my strong opinions on this subject! 1) We have a policy in place: "Data older than one month will be deleted from microscope system computers" This is on a very clear large notice at the microscope that users can not ignore. It is not our responsibility to look after users data. We have enough to do to keep the scopes running. We have a computer department to deal with data security and backups. 2) We have local disk data storage on a separate physical disk in the microscope computer. We don't back that up. but we do back up the C: drive - operating systems and software every 6 months, in case of a system crash that leaves the scope non usable. We have and to use that backup to revive a system once or twice. We use Acronis and Norton Ghost for that. 3) We have a institute file server system, where every user has plenty of space to put their files. This is big infrastructure, but very much work the effort. It is all backed up to tape by a robot every night, and data is pretty secure. I recommend that people save data DIRECTLY to their fileserver space, instead of the local hard disk. Then they have the data where they want it already and its backed up automatically, and it doesnt fill up the local disks, and they dont waste potential microscopy imaging time moving files to the server. We have gigabit ethernet in house, and i have not seen a confocal laser point scanning system yet that can push data to the file server faster then our network and fileserver can handle. This might not be true for fast widefield systems with DDC cameras. 4) In order to get around the problem of the network and fileserver not being fast enough, we are investigating having a cache server with many fast solid state disks and a 10 Gigabit connection to the microscopes that need it. data on the cache server is then moved to the fileserver, as fast as it can keep up, then is removed from the cache server. This is just an idea so far. 5) We are required by law to archive all data that is generated. We are looking at OME / OMERO and in house solutions for that. Right now it is up to users to take care of that, but we could really use a systems that works. My hopes lie in OMERO. 6) All our microscope systems are running sophos antivirus. We have noticed that some systems are slowed down by it sometimes. But we must run antivirus (on read only) because all our scopes computers are able to see the internet, and people do infect the systems with viruses from USB hard disks. FACT. 7) We strongly encourage microscope software providers to use a secure platform and write their software such that it plays nice with antivirus software. This is a necessity, not an option that would be a nice thing... if they could be bothered. The fact that we have to run windows on most systems is really a problem for our IT support. They would much rather run OSX machines and linux machines only, for obvious reasons. As a community we should DEMAND that software for driving microscope systems is not locked into an insecure windows only environment, but works on a secure OS like SELinux or OSX. Micromanager exists, for driving microscopes, in a free open source and secure environment, and as a community we should put resources towards it (free doesnt mean it costs nothing to develop) 8) On a similar theme, we as a community should demand that the microscope software providers implement open standards for data formats and meta data availability. The OME TIFF standard has been around for a while, and some software supports it, but all serious software should. The fact that we have a multitude of different formats, different between each manufacturer, or even different within one company, is truly ridiculous. In a recent format war on the next generation of DVDs, a SINGLE standard won. The open source community of working very hard to support all these different formats in ImageJ and other free software. This is the LOCI bio-formats project. We must support this project as a community with resources, feedback and resolve. The open standards work of the OME project, in defining open standard data structures for images and image metadata is also very very much in need of all our support as a community. Parting shots: The reason we suffer every day from having to run Microsoft windows on microscope systems, and battle file format compatibility problems and meta data loss is because we as a community have not been smart enough in the past to tell the manufactures what we really want. Vendor lock-in is a dirty marketing strategy, but does work. We get stuck with more of the same over and over again. We should be aware of it, and fight against it. It is against our interests, as a community. We want and need open standards, microscope system flexibility and customisation and cost effectiveness. They think we want windows, because they think that's what people want, because they already know windows. some manufacturers have already figured out that this is plain WRONG. For example Applied Precision uses a Linux operating system for the Deltavision systems, and users LIKE that. System administrators like it even better! There have always ben mac users in science, and these days more and more people have macs. More and more people also have linux as their day to day operating system. There is no reason for microscopy systems software to run on windows in favour of Linux or Mac OSX. It is only the extra cost for the companies of porting or re writing the software that prevents us getting linux and Max OSX versions. Most software developers would rather use Linux as a development platform, but often are forced into windows because their bosses think that the customers want windows software. If software is written properly , it runs on all platforms .... see ImageJ or BioImageXD. Sure hardware drivers are platform specific.... but we should demand the platform we want, and demand that the data is savable in an open standard (OME-TIFF) It is up to us to demand these things. If we remain silent, then vendor lock in continues, and no one will bother adopting or helping to implement open standards. We have to FORCE the companies to do that. Some are very open to that already... usually the smaller companies. We should encourage them as much as possible. just my 2 cents Dan Begin forwarded message: > Date: Thu, 9 Jul 2009 16:51:54 -0400 > From: Mike Tighe <[hidden email]> > Subject: I.T. questions > > Would anyone be willing to share some of your Ideas on managing data = > generated on single and multi photon instruments? Do you allow users > to = > move data directly to institute server? Has anyone tried to estimate > the = > amount of data per person or per hour of usage that is generated. > How = > often do you remove data from instrument computer? Any advice would > be = > very useful! > > Thanks!! > Mike Dr. Daniel James White BSc. (Hons.) PhD Senior Microscopist / Image Visualisation, Processing and Analysis Light Microscopy and Image Processing Facilities Max Planck Institute of Molecular Cell Biology and Genetics Pfotenhauerstrasse 108 01307 DRESDEN Germany +49 (0)15114966933 (German Mobile) +49 (0)351 210 2627 (Work phone at MPI-CBG) +49 (0)351 210 1078 (Fax MPI-CBG LMF) http://www.bioimagexd.net BioImageXD http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included http://www.chalkie.org.uk [hidden email] ( [hidden email] ) |
|
|
>
> They think we want windows, because they think that's what people > want, because they already know windows. > some manufacturers have already figured out that this is plain WRONG. unfortunately in our local tender process it has clearly been stated that it should run on windows. I'm not exactly a supporter of it. fact is that a large part of the community does not care; and they get what they deserve. only problem is, the rest of us get to suffer along with them. /Johan -- -- ------------------------------------------------ Johan Henriksson MSc Engineering PhD student, Karolinska Institutet http://mahogny.areta.org | open source microscopy: http://www.endrov.net |
 
|
Dale Callaham |
Re: I.T. questions - mandated archiving question
|
|
In reply to this post by Daniel James White
Dan,
First, I completely agree with the open standards rant. The mandate should even flow from the government level, not allowing funding for devices and systems that use proprietary formats, etc. Now, you say in #1 that archiving data is a user responsibility (with which I agree), but then, in #5, that you are required by law to archive all data generated. Someone from Princeton said they are required to archive data for 7 years. I'm just curious. I can see a research institution or granting agency mandating the archiving of data, but would that be the responsibility of the PI to see that the data is archived? If they have their own instruments that would be the case. As you say we have enough to do (as underfunded facilities) providing the facilities for the imaging. Perhaps #5 was in reference to methods for archiving by the PI? Dale Daniel James White wrote: > Hi Mike, > > Good question.... it gives me chance to vent some of my strong opinions > on this subject! > > 1) We have a policy in place: "Data older than one month will be deleted > from microscope system computers" > This is on a very clear large notice at the microscope that users can > not ignore. > It is not our responsibility to look after users data. > We have enough to do to keep the scopes running. > We have a computer department to deal with data security and backups. > > 2) We have local disk data storage on a separate physical disk in the > microscope computer. > We don't back that up. but we do back up the C: drive - operating > systems and software every 6 months, > in case of a system crash that leaves the scope non usable. > We have and to use that backup to revive a system once or twice. > We use Acronis and Norton Ghost for that. > > 3) We have a institute file server system, where every user has plenty > of space to put their files. > This is big infrastructure, but very much work the effort. > It is all backed up to tape by a robot every night, and data is pretty > secure. > I recommend that people save data DIRECTLY to their fileserver space, > instead of the local hard disk. Then they have the data where they want > it already > and its backed up automatically, and it doesnt fill up the local disks, > and they dont waste potential microscopy imaging time moving files to > the server. > We have gigabit ethernet in house, and i have not seen a confocal laser > point scanning system > yet that can push data to the file server faster then our network and > fileserver can handle. > This might not be true for fast widefield systems with DDC cameras. > > 4) In order to get around the problem of the network and fileserver not > being fast enough, > we are investigating having a cache server with many fast solid state > disks and a 10 Gigabit > connection to the microscopes that need it. data on the cache server is > then moved to the fileserver, > as fast as it can keep up, then is removed from the cache server. This > is just an idea so far. > > 5) We are required by law to archive all data that is generated. > We are looking at OME / OMERO and in house solutions for that. > Right now it is up to users to take care of that, > but we could really use a systems that works. My hopes lie in OMERO. > > 6) All our microscope systems are running sophos antivirus. > We have noticed that some systems are slowed down by it sometimes. > But we must run antivirus (on read only) because all our scopes computers > are able to see the internet, and people do infect the systems with > viruses from USB hard disks. > FACT. > > > 7) We strongly encourage microscope software providers to use a secure > platform > and write their software such that it plays nice with antivirus software. > This is a necessity, not an option that would be a nice thing... if they > could be bothered. > The fact that we have to run windows on most systems is really a problem > for our IT support. > They would much rather run OSX machines and linux machines only, for > obvious reasons. > As a community we should DEMAND that software for driving microscope > systems > is not locked into an insecure windows only environment, but works on a > secure OS like SELinux or OSX. > Micromanager exists, for driving microscopes, in a free open source and > secure environment, > and as a community we should put resources towards it (free doesnt mean > it costs nothing to develop) > > 8) On a similar theme, we as a community should demand that the > microscope software providers implement open standards > for data formats and meta data availability. > The OME TIFF standard has been around for a while, and some software > supports it, > but all serious software should. > The fact that we have a multitude of different formats, different > between each manufacturer, > or even different within one company, is truly ridiculous. > In a recent format war on the next generation of DVDs, a SINGLE standard > won. > The open source community of working very hard to support all these > different formats > in ImageJ and other free software. This is the LOCI bio-formats project. > We must support this project as a community with resources, feedback and > resolve. > The open standards work of the OME project, in defining open standard > data structures for images and image metadata > is also very very much in need of all our support as a community. > > > Parting shots: > > The reason we suffer every day from having to run Microsoft windows on > microscope systems, > and battle file format compatibility problems and meta data loss is > because we as a community > have not been smart enough in the past to tell the manufactures what we > really want. > > Vendor lock-in is a dirty marketing strategy, but does work. > We get stuck with more of the same over and over again. > We should be aware of it, and fight against it. > It is against our interests, as a community. > We want and need open standards, microscope system flexibility and > customisation and cost effectiveness. > > They think we want windows, because they think that's what people want, > because they already know windows. > some manufacturers have already figured out that this is plain WRONG. > For example Applied Precision uses a Linux operating system for the > Deltavision systems, and users LIKE that. > System administrators like it even better! > There have always ben mac users in science, and these days more and more > people have macs. > More and more people also have linux as their day to day operating system. > There is no reason for microscopy systems software to run on windows in > favour of Linux or Mac OSX. > It is only the extra cost for the companies of porting or re writing the > software that prevents us getting linux and Max OSX versions. > > Most software developers would rather use Linux as a development platform, > but often are forced into windows because their bosses think that the > customers want windows software. > > If software is written properly , it runs on all platforms .... > see ImageJ or BioImageXD. > > Sure hardware drivers are platform specific.... > but we should demand the platform we want, > and demand that the data is savable in an open standard (OME-TIFF) > > It is up to us to demand these things. > If we remain silent, then vendor lock in continues, > and no one will bother adopting or helping to implement open standards. > > We have to FORCE the companies to do that. > Some are very open to that already... usually the smaller companies. > We should encourage them as much as possible. > > just my 2 cents > > Dan > > > > > Begin forwarded message: > >> Date: Thu, 9 Jul 2009 16:51:54 -0400 >> From: Mike Tighe <[hidden email]> >> Subject: I.T. questions >> >> Would anyone be willing to share some of your Ideas on managing data = >> generated on single and multi photon instruments? Do you allow users to = >> move data directly to institute server? Has anyone tried to estimate >> the = >> amount of data per person or per hour of usage that is generated. How = >> often do you remove data from instrument computer? Any advice would be = >> very useful! >> >> Thanks!! >> Mike > > > > Dr. Daniel James White BSc. (Hons.) PhD > Senior Microscopist / Image Visualisation, Processing and Analysis > Light Microscopy and Image Processing Facilities > Max Planck Institute of Molecular Cell Biology and Genetics > Pfotenhauerstrasse 108 > 01307 DRESDEN > Germany > > +49 (0)15114966933 (German Mobile) > +49 (0)351 210 2627 (Work phone at MPI-CBG) > +49 (0)351 210 1078 (Fax MPI-CBG LMF) > > http://www.bioimagexd.net BioImageXD > http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included > http://www.chalkie.org.uk > [hidden email] > ( [hidden email] ) |
|
|
In reply to this post by Daniel James White
Hello all;
Nice to know I'm not the only meany who deletes data each month after ample warning and posted notices. We save our data directly to the equipment's HD, then file transfer to personal computers (institute server is way too small for images). There are some anecdotal secondhand ideas that acquisition is compromised if a remote user accesses the shared data folder (D:/) while someone else is acquiring images (the program and OS are on the C:/ drive). Is there any truth to this? Kathy -----Original Message----- From: Confocal Microscopy List [mailto:[hidden email]] On Behalf Of Daniel James White Sent: Friday, July 10, 2009 2:20 AM To: [hidden email] Subject: I.T. questions Hi Mike, Good question.... it gives me chance to vent some of my strong opinions on this subject! 1) We have a policy in place: "Data older than one month will be deleted from microscope system computers" This is on a very clear large notice at the microscope that users can not ignore. It is not our responsibility to look after users data. We have enough to do to keep the scopes running. We have a computer department to deal with data security and backups. |
|
|
We found this true, and that's why we disabled direct transfer (via remote login) from the local hard drives to personal computers. Nowadays users transfer their images from the local hard drive to their server zone after the experiments are done; on a gigabit network this shouldn't take long. All access to confocal PCs and image analysis workstations are only possible via server login, thus the user is automatically connected to his/her server zone the whole time. We discourage saving data directly to the server due to occasional slow-down of network speed, which has been know to interfere with data integrity.
Zoltan
On Fri, Jul 10, 2009 at 6:30 PM, Kathryn Spencer <[hidden email]> wrote: Hello all; -- Zoltan Cseresnyes Facility manager, Imaging Suite University of Cambridge, UK
|
|
|
Rosemary Rosemary White CSIRO Plant Industry GPO Box 1600 Canberra, ACT 2601 ph 02 6246 5475 fx 02 6246 5334 On 11/07/09 3:51 AM, "Zoltan Cseresnyes" <zcseresn@...> wrote: We found this true, and that's why we disabled direct transfer (via remote login) from the local hard drives to personal computers. Nowadays users transfer their images from the local hard drive to their server zone after the experiments are done; on a gigabit network this shouldn't take long. All access to confocal PCs and image analysis workstations are only possible via server login, thus the user is automatically connected to his/her server zone the whole time. We discourage saving data directly to the server due to occasional slow-down of network speed, which has been know to interfere with data integrity. |
|
|
In reply to this post by Daniel James White
On Jul 11, 2009, at 7:03 AM, CONFOCALMICROSCOPY automatic digest
system wrote: > Date: Fri, 10 Jul 2009 12:39:41 +0200 > From: Johan Henriksson <[hidden email]> > Subject: Re: I.T. questions > >> >> They think we want windows, because they think that's what people >> want, because they already know windows. >> some manufacturers have already figured out that this is plain WRONG. > unfortunately in our local tender process it has clearly been stated > that it should run on windows. I'm not exactly a supporter of it. fact > is that a large part of the community does not care; and they get what > they deserve. only problem is, the rest of us get to suffer along > with them. > > /Johan I wonder why this kind of decision is taken away from the local facility leader and PIs and put in the hands of people who have less idea what is needed on the ground. Again, its up to us to be more vocal and demand what we want, and not just take what we are given. Dan Dr. Daniel James White BSc. (Hons.) PhD Senior Microscopist / Image Visualisation, Processing and Analysis Light Microscopy and Image Processing Facilities Max Planck Institute of Molecular Cell Biology and Genetics Pfotenhauerstrasse 108 01307 DRESDEN Germany +49 (0)15114966933 (German Mobile) +49 (0)351 210 2627 (Work phone at MPI-CBG) +49 (0)351 210 1078 (Fax MPI-CBG LMF) http://www.bioimagexd.net BioImageXD http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included http://www.chalkie.org.uk [hidden email] ( [hidden email] ) |
|
|
Daniel James White |
Re: I.T. questions - mandated archiving question
|
|
In reply to this post by Dale Callaham
Hi Dale,
On Jul 11, 2009, at 7:03 AM, CONFOCALMICROSCOPY automatic digest system wrote: > Date: Fri, 10 Jul 2009 07:52:19 -0400 > From: Dale Callaham <[hidden email]> > Subject: Re: I.T. questions - mandated archiving question > > Dan, > > First, I completely agree with the open standards rant. The mandate > should even flow from the government level, not allowing funding for > devices and systems that use proprietary formats, etc. > > Now, you say in #1 that archiving data is a user responsibility (with > which I agree), but then, in #5, that you are required by law to > archive > all data generated. Someone from Princeton said they are required to > archive data for 7 years. I'm just curious. I can see a research > institution or granting agency mandating the archiving of data, but > would that be the responsibility of the PI to see that the data is > archived? Yes... the PIs are the ones who get the research funding and are responsible for the data getting archived. However, as an imaging facility providing services to them, I think its a good thing to provide a consistent way of doing that, using something like OMERO or a home brew mechanism. Then everyone reads from the same song sheet, admin becomes easier, and the data is archived (instead of that just not happening.) > If they have their own instruments that would be the case. As > you say we have enough to do (as underfunded facilities) providing the > facilities for the imaging. Perhaps #5 was in reference to methods for > archiving by the PI? I think its a facility/institute job to provide working infrastructure for data archiving, and teach the users how to get the best from it. Dan Dr. Daniel James White BSc. (Hons.) PhD Senior Microscopist / Image Visualisation, Processing and Analysis Light Microscopy and Image Processing Facilities Max Planck Institute of Molecular Cell Biology and Genetics Pfotenhauerstrasse 108 01307 DRESDEN Germany +49 (0)15114966933 (German Mobile) +49 (0)351 210 2627 (Work phone at MPI-CBG) +49 (0)351 210 1078 (Fax MPI-CBG LMF) http://www.bioimagexd.net BioImageXD http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included http://www.chalkie.org.uk [hidden email] ( [hidden email] ) |
|
|
In reply to this post by Rosemary.White
Dear All-
Short answers to a whole lot of points. Disclaimers-- 1. Our own experience, your mileage may vary. 2. I've been running the Open Microscopy Environment with Kevin Eliceiri, Peter Sorger, and Ilya Goldberg, since 2000. No money involved, but our reputations are. 3. I founded Glencoe Software, Inc. along with Peter Sorger, in 2005 to take OME's open source foundations and provide customised solutions for imaging vendors and others with majr image informatics challenges. OK, our experiences, and a few clarifications. 1. We started to build a high performance data storage system in 2002 at Dundee, in collaboration with IBM. Initially, it was VERY expensive-- £400k for 7 TB. However, almost all of that cost was not for disk, but redundant backup, SAN stuff, and lots of other goodies. So it was the start of a long-term foundation. The good news is that we have scaled this to ~150 TB spinning disk (a variety of architectures) and 2 PB tape archive and backup for relatively little money. 2. This system is not perfect, and running anything like this is very expensive. Fortunately, we've found some very very talented sysadmins to run our whole informatics infrastructure. I would however argue that, at least in the UK, funding requirements require something like this (of course, they give you the requirement, but NOT the money to meet the requirement-- that's another thread). The Wellcome Trust has been very helpful in providing funds for this system. We also combined efforts with our bioinfomatics colleagues-- we (the LM types) did storage, they did compute. We tied our facilities together, and ended up providing very powerful computing and storage solutions for our users. 3. Installing and running this meant that we completely bypassed our central IT dept. We took over everything for our centre--- networking, firewall, storage, and compute. You can imagine how popular that was. 4. I don't necessarily recommend doing this-- we started in 2002, because we had to. It allowed us to be a bit more enterprising, and offer services to our proteomics colleagues. So now all data in our College is run through this facility. We've learned more than we ever wanted to about enterprise-scale file systems. The solutions available now are much more varied, cheaper and powerful than then. However, given the benefit acros the College, the folks who make the money decisions have supported us. It's expensive, but it generates value and benefit for all of our scientific colleagues. It must be done properly, by really good people. 5. IMHO (and I do not mean this to be a flame target)-- if an computer sysadmin started an imaging facility and proposed providing a service to his/her department, fully equipped with WFM, SDCM, MPM, LSCM, etc etc., there would be alot of guffaws, not least from this list. Running these facilities takes real expertise, is always an ongoing learning exercise, and takes serious investment. Exactly the same is true for informatics. It just does not make sense to skimp on this resource, and then complain when, for example, infected USB drives bring down your facility. Expecting our dedicated microscopy facility managers to become world-class sysadmins is simply a recipe for failure. In our experience, the funding bodies respect and respond to this argument, but it needs to be made. 6. File formats: OME-TIFF is now suported by a number of vendors. We will continue to develop it (we are putting SUBSTANTIAL effort into improving our support for scanning microscopy and are now actively working towards including HCS, FLIM, and EM). Any support from the community is most welcome. It is true that vendors move to support it when their customers demand it. If a standardised format is useful for you, then condition your purchases on supporting something common and open. If OME-TIFF is it, then fine. If not, then come up-- and most importantly build, document, support, build libraries for-- and use something else. Until we do this as a community, these long threads will appear on this listserver on others (and the list members with gray hair will have memories of these threads that go way back). 7. OMERO does NOT store data as OME-TIFF. If you want your analysis tools to talk to your image files on a file system, then OMERO is NOT for you. You should use whatever image analysis software recognises your file formats. Of course, there-in lies the problem-- maybe your commercial software only recoginises its own format, and normal TIFF, and if you save as TIFF, you lose all the metadata.... One problem we have with OME-TIFF is that we don't control the commercial software that writes OME-TIFF. We are aware of variant OME-TIFFs, generated by commercial software. In our view, these occur becasue of mistakes in the file writing routines used by the commercial vendors (we provide OME-TIFF writers in Bio-Formats). We have contacted these two vendors, pointed them to our on-line validator (http://validator.openmicroscopy.org.uk), and are in discussions with them on getting the problems fixed. Sadly, response to our emails is pretty limited-- we are not customers. But you are. If you've had problems reading or writing OME-TIFF, let us know. Use the forums at http://openmicroscopy.org/community. 7. OMERO. This is an open source image data management system. It is installed at ~700 sites worldwide, but probably only in serious use by users at 50-100. We know of many sites where it is managing many TBs of data. More details on OMERO at http://openmicroscopy.org. OMERO is, to use a techie term, middleware. It is a software application that provides a single generic interface to many different file formats. This interface can be used by a range of programming environments-- C++, Java, Python, Matlab, to name a few. It includes metadata databasing, search and query facilities, user admin, image rendering, remote access to data and much more. However, OMERO is not a finished product. We try as hard as we can, but we are not perfect. We recently (in our 4.0.3 release) botched support for two main confocal formats. This really PO'd some people, many of our close and dedicated supporters (and rightly so!). We spent some serious effort looking at the problems, and have completely revamped-- from the OME Data Model, through Bio-Formats, and into OMERO-- how we support scanning microscopy. We've built a whole new testing framework for image file formats. The testing system is running now (about 30 GB of image data is tested every night) and I'll update the list on this as we move towards releasing it. We are working towards providing a facility that will allow the community to upload data for testing, and then get reports on what is imported, level of metadata support etc. Obviously, this is critical for us to do as well as possible with the image formats we already support, but also to expand into new types of data (e.g., FLIM and FCS) and new domains (e.g., EM). Look for more info at the end of the summer. The OME team is not perfect. We're human, and we make mistakes. People have been talking about the problems we are working on for over a decade. The fact that the solutions are not complete and widely available means that either those who are working on solutions aren't very good, or possibly, the problem is very very hard or most likely some of both. If you care about these problems, want the solutions, one thing to do is to talk to the vendors and encourage them to work with whomever you think is doing the best work on this problem (there are many of us, many people who contribute to this list) and provide constructive feedback. The problems are not "solved", but solutions are developing rapidly. I would also say that there are good lines of communication between the various groups involved in this work (e.g., BioImageXD, ImageJ/Fiji and OME) and look for more and more integration between us. Hope that helps. Cheers, Jason ************************** Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences MSI/WTB/JBC Complex University of Dundee Dow Street Dundee DD1 5EH United Kingdom phone (01382) 385819 Intl phone: 44 1382 385819 FAX (01382) 388072 email: [hidden email] Lab Page: http://www.dundee.ac.uk/lifesciences/swedlow/ Open Microscopy Environment: http://openmicroscopy.org ************************** On Sun, Jul 12, 2009 at 3:32 AM, Rosemary White <[hidden email]> wrote:
|
|
|
In reply to this post by Daniel James White
Hi Rosemary,
That is something your IT people should even find interesting to work with you to solve. Is is something that should not happen if the network is set up right. It might be as simple as upgrading a switch in the server room, or upgrading part of the network to Gigabit. Obviously it pays off to have very good relations with the local IT people. Remember they are geeks, and love a challenge. If you frame a problem as an interesting challenge .... or even an "i bet you can t make this work" then often they jump on it because its fun for them . One day you WILL have a hard disk failure during an imaging session, and that sessions data will be lost. I does happen. Saving data directly to a place that is immediately mirrored in 2 physical disks and also backed up to tape each night is the smart solution. You should not have to wast time on microscope systems that could be used for imaging, simply for data transfer. If the files are small, then not a big deal... but if they are large and numerous you would have users wasting hours of microscopy time every month just moving data around. That slows science down. cheers Dan On Jul 12, 2009, at 7:03 AM, CONFOCALMICROSCOPY automatic digest system wrote: > > Date: Sun, 12 Jul 2009 12:32:47 +1000 > From: Rosemary White <[hidden email]> > Subject: Re: I.T. questions > > --B_3330246775_67752070 > Content-Type: text/plain; charset="ISO-8859-1" > Content-Transfer-Encoding: quoted-printable > > Yes, we find this true also. We have little control over IT > arrangements a= > t > our institution, so if the cross-network flow is slowed or > interrupted for > any reason by IT rearranging things, the image capture software > crashes and > you tend to lose unsaved data. It=B9s easier to just save directly > to local > hard drive and copy across after a session. > Rosemary > > Rosemary White > CSIRO Plant Industry > GPO Box 1600 > Canberra, ACT 2601 > > ph 02 6246 5475 > fx 02 6246 5334 > > > > On 11/07/09 3:51 AM, "Zoltan Cseresnyes" <[hidden email]> wrote: > >> We found this true, and that's why we=A0disabled direct transfer >> (via remot= > e >> login)=A0from the local hard drives to personal computers.=A0 >> Nowadays users >> transfer their images from the local hard drive to their server >> zone afte= > r the >> experiments are done; on a gigabit network this shouldn't take >> long.=A0 All >> access to confocal PCs and image analysis workstations are only >> possible = > via >> server login, thus the user is automatically connected to his/her >> server = > zone >> the whole time.=A0 We discourage saving data directly to the server >> due to >> occasional slow-down of network speed, which has been know to >> interfere w= > ith >> data integrity.=20 >> =A0 >> Zoltan >> =20 >> On Fri, Jul 10, 2009 at 6:30 PM, Kathryn Spencer <[hidden email] >> > w= > rote: >>> Hello all; >>> =A0 =A0 =A0 =A0Nice to know I'm not the only meany who deletes >>> data each month a= > fter >>> ample warning and posted notices. >>> =A0 =A0 =A0 =A0We save our data directly to the equipment's HD, >>> then file transf= > er to >>> personal computers (institute server is way too small for images). >>> There= > are >>> some anecdotal secondhand ideas that acquisition is compromised if >>> a rem= > ote >>> user accesses the shared data folder (D:/) while someone else is >>> acquiri= > ng >>> images (the program and OS are on the C:/ drive). Is there any >>> truth to = > this? >>> =A0 =A0 =A0 =A0Kathy >>> =20 >>> =20 >>> -----Original Message----- >>> From: Confocal Microscopy List [mailto:[hidden email] >>> ]= > On >>> Behalf Of Daniel James White >>> Sent: Friday, July 10, 2009 2:20 AM >>> To: [hidden email] >>> Subject: I.T. questions >>> =20 >>> Hi Mike, >>> =20 >>> Good question.... it gives me chance to vent some of my strong >>> opinions on this subject! >>> =20 >>> 1) We have a policy in place: "Data older than one month will be >>> deleted from microscope system computers" >>> This is on a very clear large notice at the microscope that users >>> can >>> not ignore. >>> It is not our responsibility to look after users data. >>> We have enough to do to keep the scopes running. >>> We have a computer department to deal with data security and >>> backups. >> =20 >> =20 > Dr. Daniel James White BSc. (Hons.) PhD Senior Microscopist / Image Visualisation, Processing and Analysis Light Microscopy and Image Processing Facilities Max Planck Institute of Molecular Cell Biology and Genetics Pfotenhauerstrasse 108 01307 DRESDEN Germany +49 (0)15114966933 (German Mobile) +49 (0)351 210 2627 (Work phone at MPI-CBG) +49 (0)351 210 1078 (Fax MPI-CBG LMF) http://www.bioimagexd.net BioImageXD http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included http://www.chalkie.org.uk [hidden email] ( [hidden email] ) |
|
|
Ah, yes, in an ideal world some of this might happen, but it won't happen
here. Our IT folk, as in many places, are in an understaffed and overworked department. I don't think saving data to one place and copying to another is such a big deal, the network is pretty fast. At least you then have two copies of the data. Perhaps if we had really huge multi-TB sized data sets it'd be a problem, but not yet. Our flakiest instruments can't be networked anyway. And compared to the months and years and people power it's taken to develop and process the biological materials (plants in our case) to be imaged, the time taken for imaging and data transfer is negligible. The subsequent image and statistical analysis also takes much longer than collection of the original images. cheers, Rosemary On 13/07/09 7:44 PM, "Daniel James White" <[hidden email]> wrote: > Hi Rosemary, > > That is something your IT people should even find interesting to work > with you to solve. > Is is something that should not happen if the network is set up right. > It might be as simple as upgrading a switch in the server room, > or upgrading part of the network to Gigabit. > > Obviously it pays off to have very good relations with the local IT > people. > Remember they are geeks, and love a challenge. > If you frame a problem as an interesting challenge .... or even an "i > bet you can t make this work" > then often they jump on it because its fun for them . > > One day you WILL have a hard disk failure during an imaging session, > and that sessions data will be lost. > I does happen. > > Saving data directly to a place that is immediately mirrored in 2 > physical disks > and also backed up to tape each night is the smart solution. > > You should not have to wast time on microscope systems that could be > used for imaging, > simply for data transfer. If the files are small, then not a big deal... > but if they are large and numerous you would have users wasting hours > of microscopy time every month > just moving data around. That slows science down. > > cheers > > Dan > > > > > > On Jul 12, 2009, at 7:03 AM, CONFOCALMICROSCOPY automatic digest > system wrote: > >> >> Date: Sun, 12 Jul 2009 12:32:47 +1000 >> From: Rosemary White <[hidden email]> >> Subject: Re: I.T. questions >> >> --B_3330246775_67752070 >> Content-Type: text/plain; charset="ISO-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> Yes, we find this true also. We have little control over IT >> arrangements a= >> t >> our institution, so if the cross-network flow is slowed or >> interrupted for >> any reason by IT rearranging things, the image capture software >> crashes and >> you tend to lose unsaved data. It=B9s easier to just save directly >> to local >> hard drive and copy across after a session. >> Rosemary >> >> Rosemary White >> CSIRO Plant Industry >> GPO Box 1600 >> Canberra, ACT 2601 >> >> ph 02 6246 5475 >> fx 02 6246 5334 >> >> >> >> On 11/07/09 3:51 AM, "Zoltan Cseresnyes" <[hidden email]> wrote: >> >>> We found this true, and that's why we=A0disabled direct transfer >>> (via remot= >> e >>> login)=A0from the local hard drives to personal computers.=A0 >>> Nowadays users >>> transfer their images from the local hard drive to their server >>> zone afte= >> r the >>> experiments are done; on a gigabit network this shouldn't take >>> long.=A0 All >>> access to confocal PCs and image analysis workstations are only >>> possible = >> via >>> server login, thus the user is automatically connected to his/her >>> server = >> zone >>> the whole time.=A0 We discourage saving data directly to the server >>> due to >>> occasional slow-down of network speed, which has been know to >>> interfere w= >> ith >>> data integrity.=20 >>> =A0 >>> Zoltan >>> =20 >>> On Fri, Jul 10, 2009 at 6:30 PM, Kathryn Spencer <[hidden email] >>>> w= >> rote: >>>> Hello all; >>>> =A0 =A0 =A0 =A0Nice to know I'm not the only meany who deletes >>>> data each month a= >> fter >>>> ample warning and posted notices. >>>> =A0 =A0 =A0 =A0We save our data directly to the equipment's HD, >>>> then file transf= >> er to >>>> personal computers (institute server is way too small for images). >>>> There= >> are >>>> some anecdotal secondhand ideas that acquisition is compromised if >>>> a rem= >> ote >>>> user accesses the shared data folder (D:/) while someone else is >>>> acquiri= >> ng >>>> images (the program and OS are on the C:/ drive). Is there any >>>> truth to = >> this? >>>> =A0 =A0 =A0 =A0Kathy >>>> =20 >>>> =20 >>>> -----Original Message----- >>>> From: Confocal Microscopy List [mailto:[hidden email] >>>> ]= >> On >>>> Behalf Of Daniel James White >>>> Sent: Friday, July 10, 2009 2:20 AM >>>> To: [hidden email] >>>> Subject: I.T. questions >>>> =20 >>>> Hi Mike, >>>> =20 >>>> Good question.... it gives me chance to vent some of my strong >>>> opinions on this subject! >>>> =20 >>>> 1) We have a policy in place: "Data older than one month will be >>>> deleted from microscope system computers" >>>> This is on a very clear large notice at the microscope that users >>>> can >>>> not ignore. >>>> It is not our responsibility to look after users data. >>>> We have enough to do to keep the scopes running. >>>> We have a computer department to deal with data security and >>>> backups. >>> =20 >>> =20 >> > > Dr. Daniel James White BSc. (Hons.) PhD > Senior Microscopist / Image Visualisation, Processing and Analysis > Light Microscopy and Image Processing Facilities > Max Planck Institute of Molecular Cell Biology and Genetics > Pfotenhauerstrasse 108 > 01307 DRESDEN > Germany > > +49 (0)15114966933 (German Mobile) > +49 (0)351 210 2627 (Work phone at MPI-CBG) > +49 (0)351 210 1078 (Fax MPI-CBG LMF) > > http://www.bioimagexd.net BioImageXD > http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included > http://www.chalkie.org.uk > [hidden email] > ( [hidden email] ) |
|
|
I concur with Rosemary. Theory and practice can easily be two different
things when you are talking about a separate IT department. I would tell everyone to make sure they have their own backup, since you cannot rely on what the IT tell you-- speaking from past experience, losing 10 years of data because I once believed a federal IT dept on the theoretical backup solution that was in place for their network file server. Cheers, Jeff -------------------- Jeff M. Reece Reecent Technologies, LLC Honing the edge of quantitative microscopy" [hidden email] Toll-free: 877-217-8465 (direct line: 919-672-4681) www.reecent.com ----- Original Message ----- From: "Rosemary White" <[hidden email]> To: <[hidden email]> Sent: Monday, July 13, 2009 6:34 AM Subject: Re: I.T. questions > Ah, yes, in an ideal world some of this might happen, but it won't happen > here. Our IT folk, as in many places, are in an understaffed and > overworked > department. > > I don't think saving data to one place and copying to another is such a > big > deal, the network is pretty fast. At least you then have two copies of > the > data. Perhaps if we had really huge multi-TB sized data sets it'd be a > problem, but not yet. Our flakiest instruments can't be networked anyway. > > And compared to the months and years and people power it's taken to > develop > and process the biological materials (plants in our case) to be imaged, > the > time taken for imaging and data transfer is negligible. The subsequent > image and statistical analysis also takes much longer than collection of > the > original images. > > cheers, > Rosemary > > On 13/07/09 7:44 PM, "Daniel James White" <[hidden email]> wrote: > >> Hi Rosemary, >> >> That is something your IT people should even find interesting to work >> with you to solve. >> Is is something that should not happen if the network is set up right. >> It might be as simple as upgrading a switch in the server room, >> or upgrading part of the network to Gigabit. >> >> Obviously it pays off to have very good relations with the local IT >> people. >> Remember they are geeks, and love a challenge. >> If you frame a problem as an interesting challenge .... or even an "i >> bet you can t make this work" >> then often they jump on it because its fun for them . >> >> One day you WILL have a hard disk failure during an imaging session, >> and that sessions data will be lost. >> I does happen. >> >> Saving data directly to a place that is immediately mirrored in 2 >> physical disks >> and also backed up to tape each night is the smart solution. >> >> You should not have to wast time on microscope systems that could be >> used for imaging, >> simply for data transfer. If the files are small, then not a big deal... >> but if they are large and numerous you would have users wasting hours >> of microscopy time every month >> just moving data around. That slows science down. >> >> cheers >> >> Dan >> >> >> >> >> >> On Jul 12, 2009, at 7:03 AM, CONFOCALMICROSCOPY automatic digest >> system wrote: >> >>> >>> Date: Sun, 12 Jul 2009 12:32:47 +1000 >>> From: Rosemary White <[hidden email]> >>> Subject: Re: I.T. questions >>> >>> --B_3330246775_67752070 >>> Content-Type: text/plain; charset="ISO-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> Yes, we find this true also. We have little control over IT >>> arrangements a= >>> t >>> our institution, so if the cross-network flow is slowed or >>> interrupted for >>> any reason by IT rearranging things, the image capture software >>> crashes and >>> you tend to lose unsaved data. It=B9s easier to just save directly >>> to local >>> hard drive and copy across after a session. >>> Rosemary >>> >>> Rosemary White >>> CSIRO Plant Industry >>> GPO Box 1600 >>> Canberra, ACT 2601 >>> >>> ph 02 6246 5475 >>> fx 02 6246 5334 >>> >>> >>> >>> On 11/07/09 3:51 AM, "Zoltan Cseresnyes" <[hidden email]> wrote: >>> >>>> We found this true, and that's why we=A0disabled direct transfer >>>> (via remot= >>> e >>>> login)=A0from the local hard drives to personal computers.=A0 >>>> Nowadays users >>>> transfer their images from the local hard drive to their server >>>> zone afte= >>> r the >>>> experiments are done; on a gigabit network this shouldn't take >>>> long.=A0 All >>>> access to confocal PCs and image analysis workstations are only >>>> possible = >>> via >>>> server login, thus the user is automatically connected to his/her >>>> server = >>> zone >>>> the whole time.=A0 We discourage saving data directly to the server >>>> due to >>>> occasional slow-down of network speed, which has been know to >>>> interfere w= >>> ith >>>> data integrity.=20 >>>> =A0 >>>> Zoltan >>>> =20 >>>> On Fri, Jul 10, 2009 at 6:30 PM, Kathryn Spencer <[hidden email] >>>>> w= >>> rote: >>>>> Hello all; >>>>> =A0 =A0 =A0 =A0Nice to know I'm not the only meany who deletes >>>>> data each month a= >>> fter >>>>> ample warning and posted notices. >>>>> =A0 =A0 =A0 =A0We save our data directly to the equipment's HD, >>>>> then file transf= >>> er to >>>>> personal computers (institute server is way too small for images). >>>>> There= >>> are >>>>> some anecdotal secondhand ideas that acquisition is compromised if >>>>> a rem= >>> ote >>>>> user accesses the shared data folder (D:/) while someone else is >>>>> acquiri= >>> ng >>>>> images (the program and OS are on the C:/ drive). Is there any >>>>> truth to = >>> this? >>>>> =A0 =A0 =A0 =A0Kathy >>>>> =20 >>>>> =20 >>>>> -----Original Message----- >>>>> From: Confocal Microscopy List >>>>> [mailto:[hidden email] >>>>> ]= >>> On >>>>> Behalf Of Daniel James White >>>>> Sent: Friday, July 10, 2009 2:20 AM >>>>> To: [hidden email] >>>>> Subject: I.T. questions >>>>> =20 >>>>> Hi Mike, >>>>> =20 >>>>> Good question.... it gives me chance to vent some of my strong >>>>> opinions on this subject! >>>>> =20 >>>>> 1) We have a policy in place: "Data older than one month will be >>>>> deleted from microscope system computers" >>>>> This is on a very clear large notice at the microscope that users >>>>> can >>>>> not ignore. >>>>> It is not our responsibility to look after users data. >>>>> We have enough to do to keep the scopes running. >>>>> We have a computer department to deal with data security and >>>>> backups. >>>> =20 >>>> =20 >>> >> >> Dr. Daniel James White BSc. (Hons.) PhD >> Senior Microscopist / Image Visualisation, Processing and Analysis >> Light Microscopy and Image Processing Facilities >> Max Planck Institute of Molecular Cell Biology and Genetics >> Pfotenhauerstrasse 108 >> 01307 DRESDEN >> Germany >> >> +49 (0)15114966933 (German Mobile) >> +49 (0)351 210 2627 (Work phone at MPI-CBG) >> +49 (0)351 210 1078 (Fax MPI-CBG LMF) >> >> http://www.bioimagexd.net BioImageXD >> http://pacific.mpi-cbg.de Fiji Fiji is just ImageJ - Batteries Included >> http://www.chalkie.org.uk >> [hidden email] >> ( [hidden email] ) |
«
Return to Confocal Microscopy List
|
1 view|%1 views
| Free forum by Nabble | Edit this page |