In an issue very similar to last week’s outage, our shared hosting server was hit by a compromise which caused high volumes of malicious e-mail messages to be sent out. Unfortunately, this time multiple customer websites were confirmed as simultaneously compromised. All of them were using the WordPress content management system (last week’s victim was not), and may have been attacked through the same scripting vulnerability.
As such, we have audited every WordPress-based site on the server, finding a total of seven sites with malicious code additions. Those websites have been temporarily suspended while we work with their owners to re-secure the sites and remove the malicious code. We will additionally be contacting every single customer running a WordPress-based site to make certain that all installations have had the most recent security patches applied.
One of the effects of the attack is that large volumes of spam were once again sent out from our shared hosting server, causing it to be placed on e-mail blacklists and watchlists. Outbound messages are correctly passing through our server, but the reputation issues are causing them to be rejected at the recipient’s end. This affects customers with e-mail addresses at spiralEmail.com, sierraEmail.com and custom hosted domains. We have requested removal from every blacklist which allows manual delistings, but restoration of full email delivery may require 24 hours or more for automated listings to expire.
A problem with the physical hardware running our cloud database server caused user data for nccn.net accounts to be briefly unavailable on Thursday evening, January 19. This meant that users were unable to log in to their e-mail, either directly via our mail servers or via SquirrelMail at webmail.nccn.net.
(No e-mail was lost, and e-mail delivery continued uninterrupted: only sign-ins were affected.)
Our cloud provider has notified us that the affected server has been migrated to new hardware, and we have verified data integrity and restored e-mail access. If you are still having problems, please call our office at (530) 478-9822 and we can give you individual support.
Customers with e-mail addresses at spiralEmail.com, sierraEmail.com and custom hosted domains are experiencing issues today while sending outbound messages. Some messages are coming back bounced, and some are not reaching their recipient.
The root of the problem is that a website on our shared hosting server was compromised and sent out high volumes of malicious messages. This caused our shared hosting server to be placed on multiple e-mail blacklists. Messages are correctly passing through our server but the reputation issues are causing them to be rejected at the recipient’s end.
We are in the process of re-securing the server so we can initiate the process of blacklist removal. It is difficult to give a specific ETA due to the nature of blacklist removal, but the server security audit that allows us to initiate the removal process is almost complete and typically normal mail delivery resumes within 24 hours.
Edit, 12:30 p.m.: The removal request has been submitted to the primary blacklist causing delivery problems, and they estimate a 4-hour propagation window for the changes. We are reviewing the server’s mail queue to see what other blacklists need to be addressed.
Edit, 1/19: While the blacklist removal was completed within minutes of the request above, caching issues meant that some servers were using the old blacklist information for several hours (in one case up to 24 hours) after the removal was validated. We have been monitoring over the past two days and at this point it appears that all e-mail traffic is back to normal.
The storms hitting California this month are predictably creating infrastructure challenges — though at a level we haven’t seen for many years. High winds bring down trees (and power/telephone lines with them), heavy rains soak equipment (causing electrical shorts through poorly maintained weatherproofing), and massive flooding creates mobility and accessibility problems.
We’ve been informed by AT&T that they are in a statewide state of emergency, and are dispatching technicians to address known problems on an as-available basis. What this means is that if your telephone line is down, or if your DSL service is suffering problems caused by line-related issues, in some cases they are not even providing ETRs (Estimated Time of Repair), or if they are, the repair date is multiple weeks out. They are also refusing to accept any escalation requests for repairs unless the lack of service creates a medical emergency.
If your DSL service is having problems, you may be in for a long period of downtime.
Note that if a weather-related outage disrupts BOTH your Spiral DSL AND your AT&T phone line, contact AT&T’s 24-hour support at 1-800-288-2020 and report the telephone issue; in almost all cases, this will result in the fastest possible repair of both services as they address the physical damage. (Spiral can report DSL issues for dispatch of a DSL technician, but if the damage affects telephone service as well, often they will need to leave and reschedule the repair with a wiring technician.)
If your telephone service is working well and your Internet is down, give our office a call at (530) 478-9822 and we can help you diagnose the issue — DSL equipment can also be sensitive to power outages, and we can help you solve internal equipment problems on a MUCH shorter timeline.
For those of us whose service is still working, the best thing to do is hunker down and wait out the storms, but give us a call if anything needs improvement and we’ll work with you to see what can be done.
Web hosting customers, and e-mail customers with spiralemail.com and sierraemail.com addresses, are reporting problems with delivery of outbound messages. Many messages are failing to be delivered, with a delivery report being sent back to the sender indicating that they were rejected due to spam violations.
This is due to the hosting server being placed on spam blacklists after a series of accounts were compromised (in unrelated incidents) and malicious messages sent out from each of them. The accounts have been re-secured, but the process of getting removed from the blacklists is taking longer than expected due to multiple violations within quick succession.
We are actively working on restoring full operability, but do not have an ETR at this time.
Edit: Deliverability was restored Sunday evening, July 3. If you are still having problems please contact our technical support at (530) 478-9822, Ext. 1.
An unresolvable disk error on our Barracuda spam filtering server required a physical drive replacement on the evening of Monday, April 25, and the machine had to be taken offline while spam filtering settings were restored from backup. Inbound e-mail to Spiral-hosted e-mail addresses is being temporarily halted while the spam filtering server is being brought back online, and our ETR is approximately 9 p.m.
Update, 10 p.m.: Emails are again being delivered after a slightly longer than anticipated restore process. Messages that were sent during the downtime should generally be resent by the originating server and arrive in the next 1-4 hours.
Due to a disk error that halted services on one of our mail servers, Spiral customers with NCCN.net e-mail addresses found themselves unable to log in starting the evening of Thursday, March 31.
The error escaped our standard 24/7 monitoring due to a misconfiguration in the monitoring tools on that server, but was identified shortly after the start of business hours, on April 1, and the server was restored to full functionality at approximately 10:00 a.m. This outage led to high call volume causing difficulties in speaking with our technical support personnel, which has also been addressed.
Messages sent to NCCN.net customers during that time period may have been returned to sender as undeliverable. If you are missing important e-mail from the outage period, contact the sender to have them attempt delivery again, or give us a call at (530) 478-9822, Ext. 1 for tech support, and we can manually add the bounced message to your mailbox.
We’re sorry for the interruption of services, and are already discussing how our response next time can be better. It’s frustrating that this problem coincided with the April 1 observance of April Fool’s Day, but please rest assured that we take very seriously our customers and our services, and that the timing was an unfortunate coincidence.
Starting at approximately 10:45 am Monday, March 14, Internet service for our DSL customers dropped out in a wide area of Nevada County. We have received reports of service problems from around Nevada County (including but not limited to Grass Valley, Nevada City, and North San Juan), and two reports suggesting service is down as far afield as Lake Tahoe and Sacramento. We are still attempting to assess the full scope of the problem and escalate it to the phone company, and will update as soon as more information is known.
Update, 11:47 am – Our upstream provider has confirmed an issue with their transit carriers, and they are currently working on resolving the issue. Service is out as far as Modesto, so this is definitely a wider Northern California problem.
Update, 12:13 pm – Several reports from around Nevada City that service has been restored. If you were affected by today’s outage and your service is still down, please try turning off your DSL modem and then turning it back on again. If that does not restore access, please call our office at (530) 478-9822 so we can investigate and update further.
Update, 12:27 pm – Our upstream provider has confirmed a fix.
A website on our secondary hosting server was compromised today in a way that caused the site to consume all available disk space. This affected web page access and e-mail access for all sites on the server.
We have been addressing the issue for most of the morning. Web and e-mail service should now be restored for all affected hosted sites. Please call us at (530) 478-9822 if your Spiral-hosted website is still experiencing problems.
As of Wednesday morning, Jan. 20, dial-up customers are unable to place calls to our provided dial-up access numbers. Picking up the phone and calling one of those numbers manually results in hearing an “all circuits are busy” or “this call cannot be completed” message.
This has been confirmed as an outage with our upstream provider, who is currently working to restore access to those modem pools statewide. As of 10 AM we do not have an ETR on the issue but are closely monitoring it for progress.
Edit: As of 12:30 p.m. our upstream provider has cleared the outage.