Service degradation due to broader Internet issues

Today, Nov. 6, we’ve been getting scattered customer reports of difficulties accessing Spiral services. We have been impacted internally as well, with poor quality on our Voice Over IP phone lines and reduced connectivity to our Barracuda spam filtering server for inbound e-mail.

Today’s connectivity issues have been linked to nationwide outage reports by Internet backbone company Level3:

Mostly, the outages appeared to affect customers using Comcast cable internet, as Level3 is a primary provider of their backhaul.  However, due to the distributed nature of cloud computing services and Level3’s role as a major Internet fiber backbone, some issues (such as ours) affected even non-Comcast customers.

As of approximately noon PST, Nov. 6, most of the issues appeared to be resolved at Level3’s end, and all of our servers are operating normally.  We will continue to monitor the situation.

Posted in Unplanned Outages | Comments Off on Service degradation due to broader Internet issues

Major vulnerability in wi-fi encryption announced

There has been a major stir this week with the announcement of the “KRACK Attacks” — an exploit against the WPA2 standard which secures the connections to virtually all wireless networks worldwide.  As this affects everyone who uses Wi-Fi for their Internet connection, including most of our customers, we have been looking into the issue.

The first rule, for now, is: don’t panic.  While the vulnerability will affect you, so far no malicious code has been released, and vendors are already working on patches to prevent future abuse.  The good news is that WPA2 itself remains secure — there’s just a trick which can force it to reuse encryption keys in ways that an attacker can exploit.  Both computer vendors and router vendors are at work on patches that prevent that reuse; if either the router or the connected computer is patched, the connection is safe.

The most important thing you can do in order to keep your Wi-Fi secure is to install the security updates put out for the operating system of your computers/phones/tablets.  (This is important because, even if you patch your own router, you want to make certain your devices are safe on other Wi-Fi networks.)  Microsoft has already released their patch.  Apple and Google are working on theirs and a fix should be released within a week or two.

Patching your wireless router is also a best practice moving forward — although the process of applying firmware updates from your router manufacturer is more difficult than just updating your computer’s OS.  The biggest benefit to updating your router is that it will provide protection to “Internet of Things” devices which may not receive security updates; if you are only using computers/phones/tablets which are patched, router patches will provide almost no additional protection.

Spiral does not sell standalone wireless routers.  However, some DSL modems that we sell do include wireless capabilities.  We are currently speaking with our vendors about firmware updates for those modems, and if Spiral sold you a wireless modem/router, we will be happy to assist you with upgrading it at the time that those patches become available.  Please call our office at (530) 478-9822, Ext. 1 for technical support, if you would like help with that process.

Posted in Tips and Announcements | Comments Off on Major vulnerability in wi-fi encryption announced

E-mail/telephone issues

Due to a network connectivity issue at our Nevada City office, as of 11 AM Tuesday, Sept. 20, we are currently experiencing problems with inbound email for all Spiral customers.  Incoming messages are unable to reach our Barracuda spam filtering system, and therefore are not being delivered to customer mailboxes.

E-mail networks are designed to retry messages which failed to send, so at the time connectivity is restored, most or all of those backlogged messages should be retried and correctly delivered. Some messages (such as password reset emails) should be retried/resent when service is restored.

The network issue is also interfering with our ability to answer phone calls to our (530) 478-9822 office number.  Our voicemail system is working and we will respond to messages as soon as possible.

Our upstream provider has noted this as part of a regional outage and given us an ETA of 5 PM for repair.

Update, 1:30 PM:  The network is currently up and e-mail messages are passing through, but as we have already seen service return and then drop out again, we should still be presumed to be under potential outage conditions.

Update, 3:30 PM: As the network has been stable for several hours, we are returning to presumptively normal operation.

Posted in Unplanned Outages | Comments Off on E-mail/telephone issues

Outbound mail delays/rejections

Early on Tuesday, Sept. 5, a compromised website on our shared hosting server began sending out high levels of spam which caused that server to be temporarily listed on e-mail spam blacklists.  The compromised website has been re-secured and we are in the process of removing the server from the blacklists.  In the meantime, outbound mail may be disrupted for customers with,, or custom domain e-mails.

Symptoms include messages not arriving at their destination (due to being queued for later sending after the blacklisting resolves), or sent mail being returned with an error stating that the message could not be delivered.

We anticipate full service being restored by the morning of Wednesday, Sept. 6.  In the meantime, failed e-mails can be resent from alternate addresses.

Our apologies for the inconvenience.  If you have further concerns, please give us a call at (530) 478-9822, Ext. 1 for customer support.

Edit, 1:00 PM Sept. 6: The Spamhaus/CBL blacklisting which caused the most problems has been removed, which should address most spam-blocks by the end of the day.

Posted in Unplanned Outages | Comments Off on Outbound mail delays/rejections

Inbound email delivery issues (resolved)

As of early this morning, Aug. 23, many Spiral customers are either not receiving inbound e-mails or else receiving them after lengthy delays (over an hour).

This issue has been traced to our Barracuda spam filtering server.  It is correctly accepting inbound e-mail, but messages are remaining in the spam filtering queue rather than being passed through to their destination.  As of 10 a.m., this has been confirmed as a known issue with Barracuda and we have an open support ticket to resolve the issue.  We do not yet have an ETA for repair.

All inbound e-mail messages are being preserved — they are currently queued up on the Barracuda server, and will be delivered as normal after the spam filtering issues are addressed.

Our apologies for the e-mail delays. If you have any concerns, you can call our office at (530) 478-9822, Ext. 1 for customer support.

Edit: As of 12:30 p.m. Barracuda has cleared the issue and all backlogged e-mail has been delivered.

Posted in Unplanned Outages | Comments Off on Inbound email delivery issues (resolved)

Inbound email delays (resolved)

At approximately 9 a.m. today (May 3), some network issues at our Nevada City location caused our Barracuda e-mail filter to stop responding to inbound message delivery attempts.  This meant that outside messages for Spiral customers were queued for later delivery rather than getting through right away.

The network issue was resolved around noon, at which point there was a large backlog of undelivered mail.  The backlog took about an hour and a half to be processed.  At this point, all undelivered messages from earlier today should have been passed through to Spiral mailboxes correctly.

If you are still having issues with inbound mail not having shown up today, give our office a call at (530) 478-9822, Ext. 1 for customer support.

Posted in Unplanned Outages | Comments Off on Inbound email delays (resolved)

AT&T strike will complicate DSL repairs

According to several media sources, 17,000 AT&T workers have walked off the job today to protest issues affecting technicians — including outsourcing, and being forced to do work outside their area of expertise.

We have confirmed this with AT&T directly, as they are now telling us that “maintenance ETRs have been delayed indefinitely” for newly reported DSL problems.  (Unfortunately, we also cannot guarantee whether or not AT&T will honor commitments for previously scheduled DSL repairs.)

The AT&T strike affects Spiral customers because our DSL service runs on the landline telephone infrastructure owned by AT&T.  When line-quality issues are reported to us, they must be referred to the phone company, who dispatches their own technicians.

The strike follows on the heels of complaints from elected officials statewide that AT&T is abandoning their landline phone network and skimping on maintenance.  California Public Utilities Commission data shows that every month for the past three years, AT&T has failed to meet state telephone quality standards — while 90% of reported outages are supposed to be repaired within 24 hours, AT&T only meets that 24-hour goal about half of the time.  (In Q4 2016, their average was 54%.)

This is one of the many reasons that we are moving forward with our fiber-optic network — where we will own the infrastructure ourselves, and be able to take accountability for its repair and maintenance, rather than relying on a bureaucracy that doesn’t care about the challenges of rural broadband access.  “Zone A” of our network has already received $16 million in grant funding from the CPUC, and we are taking preorders within that zone as we finalize our environmental impact reports and prepare to break ground.  (If you’re outside the zone, go take the survey to let us know where fiber is needed and wanted.)

In the meantime, we will provide the best service that we can to our DSL customers, given the limitations of degrading and unrepaired infrastructure.

Posted in Tips and Announcements | Comments Off on AT&T strike will complicate DSL repairs

Dial-up access issues (resolved)

We are currently investigating an apparent outage with our upstream provider of dial-up services.  Reported symptoms include getting busy signals when calling in to the phone numbers in our dial-up access pool.

Please be patient as we isolate and address the issue.  As of 4 p.m. we do not yet have an ETR.  We will report updates here.

Update, 3/3: Network issues at our upstream provider were addressed last night and tests this morning show connectivity has been restored.

Posted in Unplanned Outages | Comments Off on Dial-up access issues (resolved)

Password compromise on shared hosting server

E-mail, client and FTP passwords on our shared hosting server have been reset due to a suspected compromise of some hosting customers’ account information.

There was another attack this week on our shared web hosting server (see also previous), injecting malicious code into a Spiral-hosted website. This week’s breach, however, was definitively traced to an unauthorized FTP connection using the account owner’s credentials.  Forensic re-review of several of the previous break-ins suggests that that method was employed for previous weeks’ website alterations, and the data available strongly suggests that multiple user passwords have been compromised.

Further review revealed that every single site affected by the recent breaches is one that had been transitioned from our secondary hosting server onto our primary server in mid-2016 when we closed the secondary server down.  The odds of that happening by chance if the breach had occurred on the current server are extremely low.  Additionally, the secondary server had far weaker security than the existing setup (a major reason for the migration).  Both of these facts point to this breach having occurred before the server transition, and old passwords from that breach being withheld for later use.

However, out of an abundance of caution we have reset ALL passwords on the current shared hosting server.

The hosting servers do NOT contain any customer financial or personal information — the only data exposed in a potential breach would be usernames, passwords, website contents and e-mail mailboxes.  However, any sensitive or personal information sent via email to accounts on that server may likewise have been compromised.

If passwords used on that server were reused elsewhere, we advise changing those as well.

Who is not affected

Customers using Spiral for internet access via DSL or dial-up are NOT affected.  Customers pre-registered for our fiber optic project are NOT affected.  Customers with e-mail addresses are NOT affected.

Who is affected

Spiral’s web hosting and custom-domain e-mail hosting customers have data on the server where we performed the password reset.  However, not all of those accounts are suspected to have been breached.

Approximately half of Spiral’s web hosting were formerly located on the secondary server where the data breach is suspected.  If you have a website hosted by Spiral, call our office at (530) 478-9822 and we can review our records and tell you whether you may have been affected.

Customers with e-mail addresses, and e-mail addresses, have mailboxes on the server where we performed the password reset.  However, those accounts were NOT ever located on the secondary hosting server believed to be compromised; at this time we have no evidence of a data breach on those accounts.

If you have concerns about your data security, or need to speak with us about your passwords, please call our office at (530) 478-9822, Ext. 1 for customer support.

Posted in Unplanned Outages | Comments Off on Password compromise on shared hosting server

DSL outage (cleared 1/31)

As of approximately 1:00 PM Tuesday, January 31, multiple and widespread reports of DSL problems have started coming in. We are investigating this as an area outage and are currently assessing the extent of the problem.

Both Grass Valley and Nevada City DSL customers are known to be affected, with reports suggesting that the outage extends beyond Nevada County. The symptoms of the outage appear to be:

• All lights on the modem indicate a normal connection to the outside world, but no IP layer traffic is getting through.
• Power-cycling your DSL equipment (turning it off and on) does not restore service.

We will update this post as we learn more.

Edit, 1:40 pm: The source of the problem appears to be a major Denial of Service attack against our upstream provider  We are coordinating with them on a response.

Edit: 2:30 pm: An ETA is not yet known.

Edit, 3:45 pm: Our upstream provider is saying that service has been restored, and we are working to verify that across our customer base.

Edit, 4:25 pm: Normal serviceability appears restored.  If your service continues to have problems, please call our office at (530) 478-9822, Ext. 1 for customer support.

Posted in Unplanned Outages | Comments Off on DSL outage (cleared 1/31)