All fileservers restored

All fileservers have been restored and all data is now available.

If you are still missing anything, please email support@engg.ksu.edu and indicate the full path of what is missing, screenshots are very helpful as well.

Engineering Data Center Offline 12/27/2018

We are having a maintenance service call on the Data Center UPS system on December 27. All servers in the Data Center will be shutdown between 8-8:30am and will remain off, most likely, all day. We hope to be able to bring them back online before 5pm, but at this time, we do not know for sure how long the maintenance will take.

This will affect the BAE and ECE research servers, the ECE Cadence server, and all College of Engineering hosted fileservers (L:, M:, N:, O:, P:, S:, T:, V:, X: and Y:).

All other services can be shifted to Seaton and are not expected to be affected by this outage.

Status Update 12/27/2018 @ 12:15pm:  Work is progressing as expected, we are still expecting to be online before 5pm.

Status Update 12/27/2018 @ 3:50pm:  UPS and Power have been restored. All services should be online now.

Fileserver Restoration Update

L: (BAE) is now complete.
N: (BAE) is now complete.
N: (ECE) is complete, minus one directory that I've communicated to that group about.

Previously complete:
I: (ECE)
K: (ChE)
M: (BAE)
S: (DOE)
T: (EEXT)
V: (Civil)
X: (HSR)
Y: (HSR)

Last Updated: 12/25/2018 @ 9:00pm

All backups have resumed

All forms of backups have been resumed, this includes servers and desktops.

File Server Restoration

V: (Civil) is now complete.
S: (DOE) is now complete.

Other drives that were already complete:
K: (ChE) is complete
T: (EEXT) is complete
X: (HSR) is complete

Y: (HSR) is complete

End of Emergency Phase

Other than file system restorations, internal infrastructure and the many straggler computers that we will be finding for months that are on the dead domain, we are essentially recovered from this event on 12/7/2018. If you find anything is still not working, please send us an email at support@engg.ksu.edu and we will look into it.

I will continue to update the File Servers post as various drive letters finish up.

Lab Printers are UP

All lab printers should be functional now. Please note that they have changed names since we have joined the Campus PaperCut system.

Please make sure you are printing to the printers that you want:

engg-duf1092-bw are the printers in the Fiedler Computer Lab in Fiedler 1092
engg-duf1093-bw are the printers in the Fiedler Learning Commons in Fiedler 1093

Departmental printers are in the same format <dept>-<room>-<printer type>

At this time, there is no printing from personal devices, you have to use our lab computers.

There is also no way to check your quota, printing to our printers does not currently consume your free quota from the University. We've set the quotas high enough that as long as you don't abuse it, you won't hit it.

Classroom/Conference room computers

Some classroom/conference room computers are now functional:

All CECS managed conference rooms and classrooms should now be complete.

Durland 1004a,1027, 1032
Engineering Hall 0093, 0096, 0097, and 1109, 1139, 3108, 3099b
Fiedler 0077, 0087, 0088, 1091, 1094, 1107, 2144
Rathbone 1044 - 1073

Last Updated: 12/7/2018 @ 5:30am

Reservation System Status

The Engineering Reservation System has been brought back online to be used for reservation of study rooms.

The Fiedler Laptops remain unavailable for the time being until our lab computer restoration effort returns them to service, at which time they will become available for reservation again.

File Server Restoration

File Server Restoration is underway.

You should start getting your drive letters mapped (L:, K:, S:, V:, etc) as you login fresh to your computers, if you have been joined to the Campus ADS. As directories complete their restoration, I then have to re-apply permissions. Once that is done, you will see those directories that you have access to. I will post a list of what has been restored on Monday.

We are still working on joing people to the Campus ADS, but most of the Faculty/Staff have been completed and we are working on graduate student computers. Faculty/Staff, if we missed you and you are now ready to be converted, or if you still have laptops that need converted, go ahead and shoot us an email to support@engg.ksu.edu and give 2 or 3 possible times that we can come by.

UPDATE: 12/5/2017 @ 7:30am: Obviously, I did not post an update on what file servers have been restored. The list would be cumbersome to write and for you to read. I have emailed Department Heads at various milestones on the restoration process and will email them again when their department is complete. Hopefully, they will pass that information along.

K: (ChE) is complete
T: (EEXT) is complete
X: (HSR) is complete
Y: (HSR) is complete

Last Update: 12/8/2018 6:00am

Lab Computers

We have started the long process to fix the lab computers.
(Note, this post will get updated instead of new posts being created for this topic.)

The following list of computers are functional, please note they are not running at peak performance/configuration. Some applications will take longer than normal to launch, others might not have all of the necessary configurations to run the way you are used to. This is simply our first attempt to getting something out there for the students.

Durland 1039, complete
Durland 1040, complete
Engineering Hall Study Rooms, 0113-0116, 1136
Engineering Hall 1113, complete
Engineering Hall 2188, complete
Engineering Hall 2189, complete
Engineering Hall 2191, complete
Engineering Hall 2192, complete
Engineering Hall 3071, complete
Fiedler 0077, complete
Fiedler Study Rooms 1082-1088
Fiedler 1091, complete
Fiedler 1092, complete
Fiedler 1093, complete, except scanner computers
Fiedler 2116, complete
Fiedler 2121, complete
Rathbone 0037, complete
Rathbone 0039, complete
Rathbone 0045, complete
Seaton 1029, complete
Seaton 1029b, complete
Seaton 1033, complete

Some Fiedler Check Laptops have returned to service in the Fiedler Learning Commons, the rest will follow later this week.
Remote Lab is now available, most nodes are available.

There is still some missing software packages and configurations, but the should be getting close to before.

Last Updated: 12/7/2018 @ 5:30am

All License Servers are UP

Mathematica code finally acquired, all license servers should be up and operation. Email support@engg.ksu.edu if you are having issues with any of them.

Status Update

Our support ticket system is finally back up. You can again email support@engg.ksu.edu for assistance. Please do not email us just to ask status of recovery, but if you are having issues that we've claimed is up, or for other issues, feel free to email us. Due to the length of time we were down, some email was lost and never delivered to us, if you emailed us over the weekend, you will need to resend those requests.

Additionally, all license servers except Mathematica and GEO-SLOPE are now online. I'm waiting on those vendors to send me new license codes before we can bring them back online.

UPDATE: 11/28/2018 @ 8:30am: GEO-SLOPE license server is online now.

Quickbooks restored.

The Quickbooks server has been restored and should be fully operational now.

However, QBReports is fully offline and will be for a week or two.

H: drive data now available

Please see our FAQ for the process to recover your h: drive data, here.

This will continue to be available for some time.

System Outage Update

We now have 16 of the 39 servers spun up and back into production. Most of these are web servers and infrastructure servers. We have several more that are spun up and getting final configurations to go into production tomorrow.

LabView and Autodesk license servers are online, as are both ChE and BAE MATLAB license servers, Minitab, ECE Comsol, ChE Materials Studio, and several others, including the ChE Aspen license server, are all up. About 70% of our license servers are now online.

The Quickbooks server is up and populated with data, but we are having some authentication issues yet, but hope to have that resolved in the morning.

The labs are still down, but we now have our intranet infrastructure online, so we will begin looking at how to fix the labs tomorrow.

File servers will begin data restoration tomorrow, but these will take several days.

Printers (other than in the labs) and copiers should be able to email their scans and work normally. If you are having problems with one, please reboot it, as it probably just hasn't picked up its new network configuration.

Services that will not be restored

1. We have decided not to recreate the ENGG Active Directory domain, but rather to join the Campus AD. Jason and Vick will be coming around to convert all computers to using that domain for authentication.

2. We will not be running our own Trent Anti/Virus server anymore. Jason and Vick will be removing our version from computers as they make their rounds for #1. By default, this will leave Windows Defender, which is usually just as reliable. If you would prefer to keep Trend, they will happily assist with installing the University version for you.

3. We will no longer be providing H: drives for personal storage. The meager quotas we were able to provide are nothing compared to a cheap USB Flash drive, or the 1TB quota that everybody has on their OneDrive. We recommend using one of those. We will have a way for people to retrieve their files from the old H: drive in a few days.
3a. For employees in the departments that we support that have been keeping work-related files on your H: drive, we will assist you in moving to using a secure folder on your departmental file server, which is really where they should have been all along.

Last Updated: 11/25/2018 @ 3:30pm

Webserver Status

The recent server outage has taken offline all webservers managed by the College of Engineering Computing Services. Webservers are in the process of being rebuilt, and following is a tentative order in which we will be rebuilding them:

1. Other

The following are fully operational to our knowledge:

1. Main college and departmental websites (all OmniUpdate)
2. Drupal 7 public websites, incl. newsletters
3. Drupal 8 public websites

The following are operational in a limited and/or read-only capacity:

1. Drupal 8 internal, incl. selective admissions and student services files
2. Research/grad PHP sites (including web-based data collection activities) -- data collection activities are fully operational, maintainers are not able to login to the server or access the database at this time.
3. CECS Intranet (reservation and checkout systems remain offline)
4. Digital Signs, incl. video wall
5. Mark Clark websites (requires further functional testing)

We will update this list as we bring servers online, and your patience is appreciated.

Last updated: 2018/11/26 9:58 PM

College of Engineering servers offline

Effectively all production servers managed by the College of Engineering Computing Services are currently offline. This includes network infrastructure, file servers, webservers, etc. I will post updates as appropriate, but don't expect anything to be working this weekend.

This effects all College of Engineering computer labs, departmental labs managed by us (BAE, ChE, CE, ECE), as well as desktops on our networks.

UPDATE: 11/24/2018 @ 12:05am: Our storage cluster is unrecoverable. We will have to rebuild servers from scratch and restore data from backups. Our backups appears to be in good shape, with the exception of the H: drives. They have been a bit behind, and it appears some of the H: drives have not been backed up in the past week. I'll know more once we get to that point.

My current priority list:

1. DHCP/DNS, and other network infrastructure services
    DHCP has been restored, desktops should now be able to get internet access
    DNS is functional from our secondary hosts provided by Central ITS
    Copiers and printers should be able to send email now.
2. Web Servers (in progress, please see above post)

3. Recording studio (This is complete, recording of classes will continue without interruption.)

4. FileServers:
    Quickbooks
    S: drive
    equivalent drives for the departments
5. Desktops migrated to new domain (started)

6. Labs (No authentication, thus they cannot be used)
7. License servers (Most license servers hosted by us are offline)
8. Print server (all College of Engineering lab printers are offline)

9. Other (yet to be identified)

Last Updated: 11/26/2018 @ 9:15am

Chilled Water Outage, started 11/20/2018 ~1pm

This outage has been resolved.

Due to a burst pipe, the chilled water service for campus is currently unavailable. This means that the Engineering Data Center currently has no air conditioning, and temperatures are extremely high. Beocat has been all shutdown, as well as all research nodes and anything not critical. If the temperatures continue to rise, we will be forced to shutdown all services. We are hopeful chilled water can be restored on Wednesday, 11/21/2018.

Update (11/22/18 @ 6:52pm): Chilled Water has been repaired, and is beginning to circulate on campus. We should be able to restore the Data Center to normal operations and bring services online tomorrow, Friday, 11/23/18.

Engineering Network disruption, 11/20/2018 8pm-10pm.

This was completed as scheduled.

Networking & Telecommunications Services will be returning our networking to its pre-Hale Library Fire status tonight. There will be intermittant network outages between the Engineering Complex building network, the Engineering network in Seaton, and wireless to the rest of the campus and the internet.