College of Engineering servers offline

Effectively all production servers managed by the College of Engineering Computing Services are currently offline. This includes network infrastructure, file servers, webservers, etc. I will post updates as appropriate, but don't expect anything to be working this weekend.

This effects all College of Engineering computer labs, departmental labs managed by us (BAE, ChE, CE, ECE), as well as desktops on our networks.

UPDATE: 11/24/2018 @ 12:05am: Our storage cluster is unrecoverable. We will have to rebuild servers from scratch and restore data from backups. Our backups appears to be in good shape, with the exception of the H: drives. They have been a bit behind, and it appears some of the H: drives have not been backed up in the past week. I'll know more once we get to that point.

My current priority list:

1. DHCP/DNS, and other network infrastructure services
    DHCP has been restored, desktops should now be able to get internet access
    DNS is functional from our secondary hosts provided by Central ITS
    Copiers and printers should be able to send email now.
2. Web Servers (in progress, please see above post)

3. Recording studio (This is complete, recording of classes will continue without interruption.)

4. FileServers:
    Quickbooks
    S: drive
    equivalent drives for the departments
5. Desktops migrated to new domain (started)

6. Labs (No authentication, thus they cannot be used)
7. License servers (Most license servers hosted by us are offline)
8. Print server (all College of Engineering lab printers are offline)

9. Other (yet to be identified)

Last Updated: 11/26/2018 @ 9:15am