[DangerousDi]

The new feature: You took too long happens if you try to vote, if you make a comments and so on. Not fun to keep going back VW.

[DarknLadyJedi]

Diego posted this yesterday on the Redclouds Forum after the massive explosion in the database errors everyone was getting:


Hello again guys!

The database switch, additions and migrations are complete.

For the more technical users:
The reason for the slowdown in performance is because we were exporting the entire database, and importing into MariaDB (this is a mysql fork), and recreating slaves.

Rough outline of the steps we took (in order).

1. Halt writes to existing cluster
2. Make complete backups
3. Import backup into new servers
4. At this time, all reads/writes were be directed towards new servers (This is when the slowdowns started during peak hours)
5. Create new slaves from the new servers. (once finished is when the slowdowns stopped. and operations were back to normal)

To sum it up:
We couldn't seamlessly do that to a live site writing to a different DB variant as the servers had to be in complete sync when taken live. There are thousands of requests and writes to the db every second and we had to have all data up to date on all servers in real time.

As I mentioned in my previous posts, we had this in progress since last Friday. Also, because of the volume of the database, and waiting on the hardware, this was a lengthy process and should have fixed the database connection concerns users were complaining about since the beginning of September. Since we have passed peak hours now, we will wait to see if the database connection issues are still there in the morning tomorrow.

Thank you for your continued patience on this matter. Things should be steady from now going forward.

[DangerousDi]

Thanks, good to know it being fixed. Always some kind of glitch with new things, huh?

[Jammer]

Does this have anything to do with the "We are down for maintenance" message I still get when I click on 'All tit flashes by date'? It's been like this for a couple of days...

[DarknLadyJedi]

Jammer, hopefully you are not still getting that message but if you are clear your cache and it should go away.

Seems several people had the problem that the maintenance page was stored in their cache and they kept getting it popping up from their cache even after maintenance was done.

[Jammer]

Yes, that seems to do the trick. Thanks DarknLadyJedi...