As far as we can tell, the new servers are working perfectly.
Have not encountered any problems at all.
These new servers give us something we have never had before, ELASTICITY.
We can finally vary the power of the servers to match the demands of increasing traffic on the site from hour to hour, day to day, week to week and month to month. In the past we had to make due with a fixed set of servers and a wildly varying set of members. In the summer we frequently see only 2,000 or so members online at any moment, while in the winter it is not uncommon to see 25,000 members online. Thats a 12x variance from one season to another.
With our new servers, tt takes appx 2 minutes for us to bring on additional resources and ramp up the processing power of the 3 primary servers that are now hosting SnoWest.
Also.. EVERYONE should notice that the latency (the time you wait for SnoWest to respond to your requests) has dropped now that we are finally in a first class tier 1 data provider.
Our data center has peering points to 16 different major back-hauls.
So if you're a Comcast customer? One hop off the network.
Verizon? One hop
AT&T? One hop
CenturyLink? One hop
All in all, we should see some significant improvements in overall performance this coming season!!
For those of you who have any interest in such things...
This is our Database Server.
Of the pair that run the forums right now, the Database server is doing most of the "Heavy Lifting".
As you can see over the last 6 hours we are cruising along at about 30% of our capacity.
Over a 24 hour period you can see the normal rise and fall of the load as members sign in and out over the day.
This last image is the IMPORTANT ONE.
This shows what this ELASTICITY we have been talking about looks like in real time.
This is the Forum server
This graph shows THREE different servers.
One the far left was the first server we loaded up and started.
We ran that one for 2 days to establish a baseline of performance.
We were burning about 45% of capacity with occasional spikes to 100%
Then we flipped the switch for the first time and brought on a bigger server and you can clearly see where the third cluster of spikes only reaches to about 65% and the small spikes before it contracted.
Then following that third spike, we brought on an even BIGGER server and took the spikes from 65% down to 35%, and really hammered the normal load down to about 10%.
And that is what we plan on doing as we roll into the fall.
As the daily load on SnoWest begins to climb this Fall & Winter, we are now able to adjust our server configurations in REAL TIME (about 2 minutes) to double or quadruple our processing power !
Gone are the days of SLOWEST.:face-icon-small-hap:face-icon-small-hap
Our new servers are no longer in Idaho Falls at a Tier 3 provider.
We have moved to the west coast into a Premier Tier 1 National Data Center with direct Fiber Optic Peer-to-Peer connections to the major ISPs in the western USA
Those RED lines show the major direct Peer-to-Peer connections to our servers.
In addition to those are a huge network of Tier-2 connections covering all of North America.
As it is, there shouldn't be a single SnoWest user that doesn't see better over all performance of the forums now.
Some will absolutely see more improvement than others, but all should see an increase.
Our new servers are somewhat nicer than before.
(snpashot inside the datacenter we moved into)
Some of you may have noticed that the forums have been down on and off over the last couple days.
We are still VERY much in the process of fine tuning these new servers and trying hard to determine EXACTLY which combination gives us the very best over all performance.
Right now SnoWest is running on 3 different servers.
Yesterday we were adjusting the primary "Forum" server.
That adjustment cause the "Database" server to go into nuclear meltdown for about 30 minutes. We doubled it's capacity to reduce it's load and all seemed well. Then, several hours later it went into Meltdown mode again. Spiked to 100% capacity and stayed their for 35 minutes.
We corrected the load problem, but couldn't really identify WHAT triggered it in the first place.
So today we started hunting, and that resulted in the forums being offline for about an hour. What we found came as a huge unexpected surprise. Those of you that are PREMIUM MEMBERS on SnoWest know that at the bottom of each thread is a listing of every member that has read that thread. Well, much to our surprise that table that stores that information just broke through 1 BILLION records. And that was the cause of the massive slowdown yesterday. So we are in the process of deleting all entries more than 1 year old. Over the course of the year we are adding between 200 and 500 thousand entries per day to that table.
So while I apologize for the downtime, slowly but surely we are getting everything in place for a nice smooth FAST winter!
I don't know if this is related, but my subscription notifications seem to be more irregular and slower than before the upgrade. Quite often it seems I get notified hours to a full day after I have read the actual post.