NYhost - Notice history

Thank you for your patience and understanding. All accounts have been restored on ELK-TX server. We were using 07/14/2024 weekly backup archive. That means any MySQL data created between 07/14-07/17 will be lost. Also, if you added an account or updated theme, template or script between 07/14 and 07/17, changes will be lost as well. We do not offer daily backups (never did). We offer weekly onsite, monthly offsite and yearly backups. Backup and restore service provided "as is" or "as available". If you host important data, you need to maintain own backup archive on local PC or cloud.

Known issues after data restore on ELK-TX server: 1. If you use custom DNS servers (external DNS like AWS or CloudFlare), you must update "A" record. Please contact us for correct IP via Help Center: https://help.nyhost.net/support/tickets/new or via email: help@nyhost.net. 2. There are no emails between 07/14 and 07/17. Please access old DEER-TX server. Emails can be fetched via POP3/webmail then synchronized manually. We didn't experience email downtime. 3. Custom MX server record may got stuck on DEER-TX (rare cases). Please drop a ticket and we'll update MX record for you. 3. FTP authentication issue has been resolved.

We'd like to be as transparent as possible. Below you'll find questions and honest answers regarding MySQL downtime on DEER-TX server:

1. What happened? DEER-TX server has suffered catastrophic data corruption within InnoDB MySQL storage. Since we use RAID-1, secondary disk mirrored exact same data. We've been working with our partner, cPanel, and were unable to revive MariaDB 10.5 server. To prevent further downtime, we decided to build new Dell server then restore data manually.

2. Is it possible to prevent such downtime in the future? The answer is no. All hosting servers (shared environment) are active and operate 24/7. Catastrophic events may occur once in a while. In the past 22 years we experienced 6 major outages. One outage lasted almost 6 days (data center fire). However, we never stored backups in the same data center. Weekly backups are being transferred to data center in Germany. In addition to that, we rotate monthly backups in our local office in New York. These backups are encrypted with protection key.

PLEASE GENERATE THEN ROTATE OWN BACKUPS LOCALLY (ON YOUR PC OR CLOUD). THIS SHOULD BE DONE AT LEAST ONCE PER 30 DAYS. SITES OVER 50 GB ARE NOT BEING BACKED AUTOMATICALLY! While we didn't have large websites on DEER-TX server, we see many websites over 100 GB on other reseller and SD servers. If you need any assistance with backup creation, please do let us know: help@nyhost.net. We're here to help!

3. How do you protect website data? We use SSD disks (data center edition) with basic RAID-1 protection. If one disk fail, we'll replace failed drive then update RAID-1. However, in case of software data corruption, secondary disk will get same corrupted data. That's why we're rotating weekly and monthly backups. Additionally, every 3-5 years we upgrade hardware, especially SSD disks and CPU. Software is also being REPLACED every 3-5 years. Operating systems do change all the time. Right now we're rolling major upgrades from CloudLinux 7.x to CloudLinux 9.x. In order to provide best possible protection, we do not use user/pass with SSH, free Linux software (we use commercial grade software). We also use basic software firewall and commercial grade hardware firewall from Cisco. And finally, all servers are rack-mounted 2U or 4U built by Dell or Supermicro.

4. Why it took almost 2 days to restore all data? Are you understaffed? No, we had to wait for new server in Dallas, TX (1 day) then restore 500+ cPanel accounts one by one, reset WHM permissions, verify DNS etc. We normally move accounts from one server to another during scheduled upgrades. In this case, restore would take less than 8 hours and won't cause any downtime at all.

5. Did you lose any account during this outage? Yes, we did. We lost 1 website out of 500+ because cPanel user had "disabled backup" or "excluded" setting all the time. Automatic weekly and monthly backups were not running for that particular website. We doubt that user disabled backups. It's possible that previous host disabled auto backups and those settings were copied over. We also lost 2 other websites which were added between 07/15 and 07/17, however, accounts are still active at old host and transfer/restore can be initiated again.

6. Are you going to improve data restore times in the future? Certainly. First of all, we'll continue replacing old server hardware every 3-5 years. More of this, we'll keep spare server or two just in case. As you may noticed, we do not host thousands of websites on single server (like others do). There is a reason for this: less users = less impact. Each Dell server can host over 1K accounts easily. We cap user per server limit to 500 (at our own expense).

7. Are you planning to compensate? Yes, because we offer 99.9% uptime guarantee with most web hosting plans. SLA credits will be issued in September.

Thank you very much for your patience during data restore. We'll do everything possible to keep your websites up and running 24/7 on feature rich professional servers. If you have any questions or concerns, please drop a ticket to NYhost help center: https://help.nyhost.net/support/tickets/new . You can also email: help@nyhost.net at any time. If you wish to expedite any ticket, please contact admin@nyhost.net. Thank YOU!

We've restored over 50% of all data on ELK-TX server. Once we complete data restore, we'll install Softaculous App installer. Thank you again for your patience.

New server ELK-TX is ready for data restore (07/14/2024 weekly backup). We're now processing accounts which must be restored ASAP (please submit ticket with specific cPanel user or simply email help@nyhost.net). There is no ETA, but most accounts will be up by tomorrow morning. Once we complete restore, we'll share complete investigation report, FAQ and improvement plan for faster data recovery in the future. Thank you again for your patience!

DEER-TX server update 👍 New Dell server has been received in Dallas, TX. New server ID is ELK-TX. We're currently setting up new CloudLinux 9 OS then cPanel with MariaDB 10.5. ETA: 4 hours. Data restore (per cPanel account) will follow. Old server (DEER-TX) will remain active for about 10 days. If you wish to speed up restore process, it's time to tell us which website to restore first. Please submit a ticket to Help Center: https://help.nyhost.net/support/tickets/new Thank you for your patience.

New Dell server is enroute to Dallas data center. It should be received by tomorrow evening, July 18th or July 19th latest. Once we setup new server, we'll load CloudLinux 9 Pro OS then cPanel. After that backups will be moved from old DEER-TX server. We're able to keep old server up regardless of MySQL downtime. Email, DNS and cPanel backups are NOT affected. We'll restore websites from letter "A" first. All websites are important but If you need to restore particular website faster, you'll be able to do so later on. New server ID will be ELK-TX. DNS will remain the same. Server IP will change, but we're going to save/transfer old IP just in case you need it (for example, website is connected via "A" record externally). Downtime SLA credits will be issued in September. Once again, we do apologize for the inconvenience. We're working hard to restore all data ASAP. If you have any questions or concerns, please don't hesitate to contact us: help@nyhost.net or call 1-646-595-0555 at any time. We'll get back to you at once. Thank you!

Unfortunately, file system check didn't find any issues with SSD disk. Entire MariaDB partition got corrupted somehow. cPanel admins were unable to fix MariaDB and innodb storage issues. Server upgrade was scheduled for August 20th. As you may already know, cPanel stopped supporting CentOS in June and "DEER-TX" hardware isn't optimal (it was online since 2021). Normally, we'd setup new server then move accounts in live mode without downtime from one server to another. Right now we have to bring new server up, load new CloudLinux 9 then move backups over. After that we can restore accounts one by one. It take at least 24-48 hours to restore all data. Accounts with letter "A" will be restored first. If you need to restore specific account faster, you'll be able to do so once we bring new server up. It should be up by 07/18 18:00 EST.

We're currently performing another file system check on MySQL partition. ETA: 1-2 hours. Once we complete this, we'll reboot DEER-TX server. Thank you for your patience.

MySQL partition got corrupted somehow. We'll run another FSCK on primary disks then reboot. Thank you again for your patience.

DEER-TX server is up but in unhealthy state. It may go down at any time. There are multiple errors on primary SDA disk and we'll try to correct it with another FSCK (it will be scheduled) then reboot again. Backups generated 07/14/24 are not affected and stored locally in Dallas TX. We also have weekly and monthly backups on remote server in New York and Wiesbaden, Germany. We do apologize for the inconvenience.

We are currently running a file system check on primary SSD disks in raid-1 array. There is no ETA yet. We do apologize for the inconvenience.