A series of events last night caused approximately 150 virtual machines to go offline, including our portal ServerPointAPPS.com.
These virtual machines’ disks are hosted on SAN #3, a SAN that lost its failover approximately six weeks ago. The failover couldn’t be established after that event due to flaws in the SAN’s software.
Since then, we’ve been working on preparing new SAN technology to transfer clients over to it. We expect to start doing this within two weeks.
SAN #3 went offline at approximately 8PM PST (GMT -8) last night, due in part by an accidental fiber cut from an electrician installing new power circuits in our data center.
Resolution: we are working on taking this SAN (SAN #3) offline over the next few weeks by transferring clients over to a brand new one.
We have also created new programming code that will prevent data corruption on VMs. VMs are supposed to go offline when this happens, and then start gracefully when the SAN comes online. This did not happen, and VMs were left running without their disks.
This bug has now been corrected.
Unfortunately, about six clients were affected by this and we are working with them on this. If your VM has any issues, please contact us immediately.