An unresolvable disk error on our Barracuda spam filtering server required a physical drive replacement on the evening of Monday, April 25, and the machine had to be taken offline while spam filtering settings were restored from backup. Inbound e-mail to Spiral-hosted e-mail addresses is being temporarily halted while the spam filtering server is being brought back online, and our ETR is approximately 9 p.m.
Update, 10 p.m.: Emails are again being delivered after a slightly longer than anticipated restore process. Messages that were sent during the downtime should generally be resent by the originating server and arrive in the next 1-4 hours.
Due to a disk error that halted services on one of our mail servers, Spiral customers with NCCN.net e-mail addresses found themselves unable to log in starting the evening of Thursday, March 31.
The error escaped our standard 24/7 monitoring due to a misconfiguration in the monitoring tools on that server, but was identified shortly after the start of business hours, on April 1, and the server was restored to full functionality at approximately 10:00 a.m. This outage led to high call volume causing difficulties in speaking with our technical support personnel, which has also been addressed.
Messages sent to NCCN.net customers during that time period may have been returned to sender as undeliverable. If you are missing important e-mail from the outage period, contact the sender to have them attempt delivery again, or give us a call at (530) 478-9822, Ext. 1 for tech support, and we can manually add the bounced message to your mailbox.
We’re sorry for the interruption of services, and are already discussing how our response next time can be better. It’s frustrating that this problem coincided with the April 1 observance of April Fool’s Day, but please rest assured that we take very seriously our customers and our services, and that the timing was an unfortunate coincidence.
Starting at approximately 10:45 am Monday, March 14, Internet service for our DSL customers dropped out in a wide area of Nevada County. We have received reports of service problems from around Nevada County (including but not limited to Grass Valley, Nevada City, and North San Juan), and two reports suggesting service is down as far afield as Lake Tahoe and Sacramento. We are still attempting to assess the full scope of the problem and escalate it to the phone company, and will update as soon as more information is known.
Update, 11:47 am – Our upstream provider has confirmed an issue with their transit carriers, and they are currently working on resolving the issue. Service is out as far as Modesto, so this is definitely a wider Northern California problem.
Update, 12:13 pm – Several reports from around Nevada City that service has been restored. If you were affected by today’s outage and your service is still down, please try turning off your DSL modem and then turning it back on again. If that does not restore access, please call our office at (530) 478-9822 so we can investigate and update further.
Update, 12:27 pm – Our upstream provider has confirmed a fix.
A website on our secondary hosting server was compromised today in a way that caused the site to consume all available disk space. This affected web page access and e-mail access for all sites on the server.
We have been addressing the issue for most of the morning. Web and e-mail service should now be restored for all affected hosted sites. Please call us at (530) 478-9822 if your Spiral-hosted website is still experiencing problems.
As of Wednesday morning, Jan. 20, dial-up customers are unable to place calls to our provided dial-up access numbers. Picking up the phone and calling one of those numbers manually results in hearing an “all circuits are busy” or “this call cannot be completed” message.
This has been confirmed as an outage with our upstream provider, who is currently working to restore access to those modem pools statewide. As of 10 AM we do not have an ETR on the issue but are closely monitoring it for progress.
Edit: As of 12:30 p.m. our upstream provider has cleared the outage.
Heavy snowstorms are forecast for Dec. 24-25 throughout our county — and where there’s snow, there’s often downed power and phone lines.
It’s impossible to predict what sort of outages our Christmas storm might cause, but due to the aging infrastructure here in rural California, there’s a good chance that this storm might knock out the Internet connection of hundreds, or thousands, of residents. Please rest assured that we’re monitoring for these issues … unfortunately, our ability to address them is limited, as the DSL service that we provide is delivered through lines owned by AT&T, and any outages caused by physical damage must be repaired by AT&T themselves.
If a weather-related outage disrupts BOTH your Spiral DSL AND your AT&T phone line, contact AT&T’s 24-hour support at 1-800-288-2020 and report the telephone issue; in almost all cases, this will result in the speediest repair of both services as they address the physical damage. (Spiral can report DSL issues for dispatch of a DSL technician, but if the damage affects telephone service as well, often they will need to leave and reschedule the repair with a wiring technician.)
If your telephone service is working well and your Internet is down, we will do our best to address it in a timely fashion. Call our office at (530) 478-9822 and leave us a message with details.
Please note that, in order to let our staff spend holiday time with their families, we are closing our office for Thursday 12/24 and Friday 12/25. However, we will monitor our voicemail to address urgent issues. (Standard tech support, billing, and new service inquiries will be handled Monday 12/28 when we return to the office.) Give us a call at any time at (530) 478-9822 in order to leave us a message.
Have a wonderful (and safe) white Christmas, and an equally great holiday season!
We’ve received many reports today of DSL connection problems from Spiral customers around Nevada County. Symptoms include the DSL modem showing a connection, but computers not being able to access web pages or internet content. We are investigating this as a potential area-wide outage and coordinating our investigations with our upstream providers.
Please be patient: due to high call volume, if you call our office you may not be able to speak to a technician right away, but we are addressing the issue as quickly as possible.
As of 11:45 AM, the problem with at least some of the affected customers has been traced to a DNS issue and we are working on our DNS servers to address the problem.
Update: As of approx. 12:15 pm, the DNS issue should be cleared.
Update: There appears to be a second issue related to downed lines yesterday in the Nevada City area. AT&T has already worked on those repairs, and for affected customers the current solution is to reset your DSL circuit by turning your modem off and on. Please call our office at (530) 478-9822 if that does not restore your service, as we might need to perform a circuit rebuild while your DSL modem is turned off.
After several long years, the California PUC has just voted to fund our grant application for Zone A of our Nevada County fiber optic project. We cannot put into words how excited we are to be bringing next-generation speeds to Nevada County, and to be building our own rock-solid infrastructure that we can rely on rather than AT&T’s aging phone lines.
The next few years will see some major changes. To see project maps for current and future fiber deployment, and to register your interest, go to spiral.com/fiber and enter your street address. Also be sure to check out our Nevada County broadband issues blog at www.NevadaCountyConnected.com.
We have received confirmation from the phone company that this weekend’s outage affecting Nevada City area DSL and phone lines has been cleared.
If your DSL service is not yet working, power-cycle your DSL modem (unplug its electrical cord, and once its lights are off, plug it back in). If that doesn’t fix your problem, give our office a call at (530) 478-9822, Ext. 1 for customer support.
Some of our DSL customers are still experiencing Internet service that is unavailable. We were told that AT&T was dispatching a repair truck sometime today (Sat July 4) between 4pm and 8pm. This is no guarantee that service will become available again. When the service is eventually turned on, you will need to unplug and re-plug the power cord (and only the power cord) on your modem. If all the lights turn green again, you’re good to go. If not, that means that service is still unavailable. You will want to test this at a later time as AT&T does not notify us when circuits are back up. Because that is how AT&T chooses to communicate, by not doing so at all. Please be patient and, above all, enjoy the 4th of July holiday; being aware that there is a high fire danger.
UPDATE: Based on customer conversations and the geographic distributions of affected customers, it looks like the outage may be related to a fire in the North San Juan area on Thursday night 7/2. We are hearing reports from affected customers that phone service was restored in various areas either Friday 7/3 or Saturday 7/4 (some say phone service was not interrupted at all), but DSL remains down for all affected. However, AT&T has identified circuits as far away as the Banner Mountain area as affected by the known outage.
AT&T’s maintenance operations center is closed on Sundays (though their field techs will continue to work on this issue), and so we do not anticipate having any further updates on their repairs until Monday morning.