At Snort we have an extensive amount of monitoring taking place to make sure the health of Snort.org is as optimal as we can make it.
One of the things we monitor is response time, or how long it takes, from the time your browser requests Snort.org, to the time we fulfill the entire page or whatever is being loaded. We strive for a sub-100ms response time.
We'd like to go faster, but look, this is reality, nothing is perfect, and Snort is a very complex beast.
Setting aside the millions of hits a day at Snort.org gets, lets concentrate on the people that have PulledPork and Oinkmaster checking for new rules, automatically, in a crontab. We have nearly 500,000 PulledPork requests a day, and this "GET" request is very quick. Since we generally release rule packs on Tuesdays and Thursdays, most of the people hitting Snort.org for the md5 of the rulepack, find out the md5 hasn't changed, and move on.
Unless of course, we deploy a new rule pack, that md5 changes, then you grab the full rule pack. Working exactly as intended. We love pulledpork for this, and we wish the rest of the oinkmaster users would move off of oinkmaster, as it helps us alleviate a lot of load on the server.
We use load balancing, and even Cloudflare in front of Snort.org to cache the majority of requests to the site. In fact, about 85% of the content served from Snort.org is cached.
The remainder of this traffic, for the most part, is document and rule downloads.
This only becomes a problem, basically, at the top of the hour. (Our downloaders love 12pm and 4pm the most for some reason). At every hour, we have huge spikes of traffic, caused by people running pulledpork (or, for some reason, oinkmaster) in a cron to download the ruleset on the hour.
It's perfectly fine that you do this.
However, if we can encourage, say, 10% of you, to randomize your crontab's time, even to 10 minutes past the hour, the response time on our servers would drop tremendously. (Now, don't everyone go set their crontab to 10 past the hour, it was just an example!)
Please keep in mind that no one has complained about the response time of the site, and we aren't overly concerned with the issue. We just prefer to head this off at the pass, before it becomes an issue.
We add over 1,000 new users to the site every week, and with well over 500,000 active users on Snort.org now, and we show no signs of slowing down. In fact, by all the metrics we track, activity is increasing. This is fantastic, and we love that the fact that our community is strong.
However, if we can adjust some of our crontab run times for the rule update software that you all are running, we can keep the experience as optimal as we can for everyone for a long time to come.
I appreciate you doing so, thanks a lot!
Keep Snorting!
