We moved from Hostinger to DigitalOcean
It has been 4 months since we started working on Hyvor Talk full-time. We have grown steadily and this month, we got clients with millions of traffic per month. We had no experience in handling high traffic. As a result, our servers went down when we got spikes in traffic. After trying different fixes, we finally made the decision to move from our previous host Hostinger to our new host DigitalOcean. This is our experience.
Increasing Connection and File Limits
We used Hostinger VPS Hosting as our server for Hyvor Talk – we only had one server. Apache started crashing when we got traffic spikes. First, we increased the max connections limit of Apache and MYSQL, which solved the problem for about 4 hours. After that, the server again crashed. This time we understood that the open file limit in Ubuntu causes the error as we had 100+ concurrent WebSocket connections to the server.
Our realtime system runs on Node.js and was proxied through Apache. However, Apache with mpm_prefork isn’t very good to handle concurrent connections with ease. To solve that easily, we installed Nginx on the same server and proxied the Node.js server through it. Great! We could sleep very well for another few hours.
The server again crashed when we started getting 200+ concurrent connections. Then, we increased the max connections directive of the NGINX server. Even it fixed the problem up to 1000 connections, unfortunately, we couldn’t increase the open file limit in Ubuntu as Hostinger uses OpenVZ for VPS servers. We needed a VPS server with KVM virtualization.
Even though I’m talking softly at this moment, we were extremely panicked at that moment. On the one hand, the servers are restarting in a few hours, and there were awful downtimes on our sites. On the other hand, we know that this is an extremely bad customer experience. So, we need quick solutions.
Fortunately, I had a $100 coupon from DigitalOcean and had tested it a few weeks back. I searched the internet if “DigitalOcean droplets use KVM” (Droplets are VPS servers. It’s just the name used at DigitalOcean). The answer was unclear but resources alluded that they do. I checked it by myself on a test server.
hostnamectl command to check it out. It will say what
Virtualization your server uses.
Hopefully, it was KVM!
Then, we set up the new server with NGINX and increased open file limits. Finally, we migrated the realtime system to the new server and changed the DNS records. Thankfully, everything in Hyvor Talk works fine even the realtime system totally crashed. Only thing is that comments won’t be shown in realtime. So, there wasn’t any downtime in the migration of the realtime system.
I suggest that you read this article to find out how to configure your server to handle thousands of concurrent connections without any problems. Also, follow this guide to learn how to increase open file limits on Ubuntu.
At this point, part of Hyvor Talk was on Hostinger and the other part was on DigitalOcean.
We slept well… for about 1 day 🙂
Certbot and Apache Error
We had used Certbot to deploy Let’s Encrypt SSL certificates on the server. And, however (we do not yet know how), this had changed the Apache configuration, which caused the server to run into an error when getting spikes.
Once an hour or two, the Apache server shut down. We had to manually restart it this time :(.
There were a few resources describing solutions for this problem but none of them worked. The only solution was to reinstall Apache. While we were deciding whether or not to do it, something happened.
The server went down and we couldn’t even access it via SSH. We asked Hostinger customer support about this and waited about 5 minutes – no reply. We made the decision at this point to move the whole site to a new server on DigitalOcean.
Moving to DigitalOcean
First, we sent an email to our users regarding this issue (with the fear in our hearts that they will leave us due to multiple downtimes we had and this huge one).
It took about 1 hour to completely set up the server, install packages, and pull the source codes and set up deployment (our fear pushed us to do it fast). And, still, our original server on Hostinger didn’t function, and after about 30 minutes, we got a reply from Hostinger that we have gone beyond the CPU limits and the server was suspended. Whoah! We made a good decision to move. However, an operator at Hostinger could cancel the suspend and allow us to log in via SSH.
Then, we migrated our database from Hostinger to DigitalOcean. The correct way to do is by using MYSQL replication. But, our servers are already down. So, we simply dumped the databases on Hostinger and imported it to a managed database cluster in DigitalOcean.
We manually checked if all the data has migrated. Everything was fine at the moment. Then, we changed our DNS records to the new server in Cloudflare. Yay! Our servers are again up! It took us about 2.30 hours for the complete process.
The only problem we found was a character encoding problem. Worst, we didn’t notice it until going live with the new server. We’ll write about how we fixed it in another article.
When using Hostinger we had only one server for both Apache and MYSQL. It wasn’t scalable. We always needed to keep these two separated so that we can horizontally scale (add more machines to handle the traffic) our servers in the future.
In DigitalOcean, this is quite easy. We can access everything in one clean dashboard. AWS (Amazon Web Services) is also fine but I personally think it is overly complex and never liked their interface (we still use Amazon S3 for external backups 😊).
Managed vs Unmanaged MYSQL Database
For the MYSQL database, we chose Managed Databases Clusters over setting up a droplet and installing MYSQL. We weren’t sure if this is the best thing to do. However, now we think we made a fine decision. Even the managed databases are expensive, they offer statistics and more insights. We can see long-running queries and fix them. And, we can easily add backup nodes with a few clicks.
However, there are two cons I noticed.
- You cannot change MYSQL system variables.
- You cannot scale down clusters ones they are scaled up.
DigitalOcean also offer managed Redis. However, we set up Redis in the server when migrating. For now, it is stable and does a good job by handling cache using less RAM.
Moving from Hostinger to DigitalOcean was an “instant decision”. However, after a week, we feel that we made the right decision at the right moment. Hostinger is a great service, which we have used for about 1 year. We still use them to host Hyvor Groups. However, their VPS servers are limited to 8GB, and when you service scale, you’ll surely need to consider another solution.
This migration was extremely stressful for us. We feared that users will leave us, that didn’t happen. Instead, we received replies to our emails encouraging us (which we didn’t expect).
If you have any questions to ask, please comment below.