Thursday, November 29, 2018

Lab Computers

We have started the long process to fix the lab computers.
(Note, this post will get updated instead of new posts being created for this topic.)

The following list of computers are functional, please note they are not running at peak performance/configuration. Some applications will take longer than normal to launch, others might not have all of the necessary configurations to run the way you are used to. This is simply our first attempt to getting something out there for the students.

Durland 1039, complete
Durland 1040, complete
Engineering Hall Study Rooms, 0113-0116, 1136
Engineering Hall 1113, complete
Engineering Hall 2188, complete
Engineering Hall 2189, complete
Engineering Hall 2191, complete
Engineering Hall 2192, complete
Engineering Hall 3071, complete
Fiedler 0077, complete
Fiedler Study Rooms 1082-1088
Fiedler 1091, complete
Fiedler 1092, complete
Fiedler 1093, complete, except scanner computers
Fiedler 2116, complete
Fiedler 2121, complete
Rathbone 0037, complete
Rathbone 0039, complete
Rathbone 0045, complete
Seaton 1029, complete
Seaton 1029b, complete
Seaton 1033, complete

Some Fiedler Check Laptops have returned to service in the Fiedler Learning Commons, the rest will follow later this week.
Remote Lab is now available, most nodes are available.

There is still some missing software packages and configurations, but the should be getting close to before.

Last Updated: 12/7/2018 @ 5:30am

All License Servers are UP

Mathematica code finally acquired, all license servers should be up and operation. Email support@engg.ksu.edu if you are having issues with any of them.

Tuesday, November 27, 2018

Status Update

Our support ticket system is finally back up. You can again email support@engg.ksu.edu for assistance. Please do not email us just to ask status of recovery, but if you are having issues that we've claimed is up, or for other issues, feel free to email us. Due to the length of time we were down, some email was lost and never delivered to us, if you emailed us over the weekend, you will need to resend those requests.

Additionally, all license servers except Mathematica and GEO-SLOPE are now online. I'm waiting on those vendors to send me new license codes before we can bring them back online.

UPDATE: 11/28/2018 @ 8:30am: GEO-SLOPE license server is online now.

Quickbooks restored.

The Quickbooks server has been restored and should be fully operational now.

However, QBReports is fully offline and will be for a week or two.

H: drive data now available

Please see our FAQ for the process to recover your h: drive data, here.

This will continue to be available for some time.

Monday, November 26, 2018

System Outage Update

We now have 16 of the 39 servers spun up and back into production. Most of these are web servers and infrastructure servers. We have several more that are spun up and getting final configurations to go into production tomorrow.

LabView and Autodesk license servers are online, as are both ChE and BAE MATLAB license servers, Minitab, ECE Comsol, ChE Materials Studio, and several others, including the ChE Aspen license server, are all up. About 70% of our license servers are now online.

The Quickbooks server is up and populated with data, but we are having some authentication issues yet, but hope to have that resolved in the morning.

The labs are still down, but we now have our intranet infrastructure online, so we will begin looking at how to fix the labs tomorrow.

File servers will begin data restoration tomorrow, but these will take several days.

Printers (other than in the labs) and copiers should be able to email their scans and work normally. If you are having problems with one, please reboot it, as it probably just hasn't picked up its new network configuration.

Sunday, November 25, 2018

Services that will not be restored

1. We have decided not to recreate the ENGG Active Directory domain, but rather to join the Campus AD. Jason and Vick will be coming around to convert all computers to using that domain for authentication.

2. We will not be running our own Trent Anti/Virus server anymore. Jason and Vick will be removing our version from computers as they make their rounds for #1. By default, this will leave Windows Defender, which is usually just as reliable. If you would prefer to keep Trend, they will happily assist with installing the University version for you.

3. We will no longer be providing H: drives for personal storage. The meager quotas we were able to provide are nothing compared to a cheap USB Flash drive, or the 1TB quota that everybody has on their OneDrive. We recommend using one of those. We will have a way for people to retrieve their files from the old H: drive in a few days.
3a. For employees in the departments that we support that have been keeping work-related files on your H: drive, we will assist you in moving to using a secure folder on your departmental file server, which is really where they should have been all along.

Last Updated: 11/25/2018 @ 3:30pm

Saturday, November 24, 2018

Webserver Status

The recent server outage has taken offline all webservers managed by the College of Engineering Computing Services. Webservers are in the process of being rebuilt, and following is a tentative order in which we will be rebuilding them:
  1. Other
The following are fully operational to our knowledge:
  1. Main college and departmental websites (all OmniUpdate)
  2. Drupal 7 public websites, incl. newsletters
  3. Drupal 8 public websites
The following are operational in a limited and/or read-only capacity:
  1. Drupal 8 internal, incl. selective admissions and student services files
  2. Research/grad PHP sites (including web-based data collection activities) -- data collection activities are fully operational, maintainers are not able to login to the server or access the database at this time.
  3. CECS Intranet (reservation and checkout systems remain offline)
  4. Digital Signs, incl. video wall
  5. Mark Clark websites (requires further functional testing)
We will update this list as we bring servers online, and your patience is appreciated.

Last updated: 2018/11/26 9:58 PM

Friday, November 23, 2018

College of Engineering servers offline

Effectively all production servers managed by the College of Engineering Computing Services are currently offline. This includes network infrastructure, file servers, webservers, etc. I will post updates as appropriate, but don't expect anything to be working this weekend.

This effects all College of Engineering computer labs, departmental labs managed by us (BAE, ChE, CE, ECE), as well as desktops on our networks.

UPDATE: 11/24/2018 @ 12:05am: Our storage cluster is unrecoverable. We will have to rebuild servers from scratch and restore data from backups. Our backups appears to be in good shape, with the exception of the H: drives. They have been a bit behind, and it appears some of the H: drives have not been backed up in the past week. I'll know more once we get to that point.

My current priority list:
1. DHCP/DNS, and other network infrastructure services
    DHCP has been restored, desktops should now be able to get internet access
    DNS is functional from our secondary hosts provided by Central ITS
    Copiers and printers should be able to send email now.
2. Web Servers (in progress, please see above post)
3. Recording studio (This is complete, recording of classes will continue without interruption.)
4. FileServers:
    Quickbooks
    S: drive
    equivalent drives for the departments
5. Desktops migrated to new domain (started)
6. Labs (No authentication, thus they cannot be used)
7. License servers (Most license servers hosted by us are offline)
8. Print server (all College of Engineering lab printers are offline)
9. Other (yet to be identified)

Last Updated: 11/26/2018 @ 9:15am

Tuesday, November 20, 2018

Chilled Water Outage, started 11/20/2018 ~1pm

This outage has been resolved.

Due to a burst pipe, the chilled water service for campus is currently unavailable. This means that the Engineering Data Center currently has no air conditioning, and temperatures are extremely high. Beocat has been all shutdown, as well as all research nodes and anything not critical. If the temperatures continue to rise, we will be forced to shutdown all services. We are hopeful chilled water can be restored on Wednesday, 11/21/2018.

Update (11/22/18 @ 6:52pm): Chilled Water has been repaired, and is beginning to circulate on campus. We should be able to restore the Data Center to normal operations and bring services online tomorrow, Friday, 11/23/18.

Engineering Network disruption, 11/20/2018 8pm-10pm.

This was completed as scheduled.

Networking & Telecommunications Services will be returning our networking to its pre-Hale Library Fire status tonight. There will be intermittant network outages between the Engineering Complex building network, the Engineering network in Seaton, and wireless to the rest of the campus and the internet.