Inbound email delays (resolved)

At approximately 9 a.m. today (May 3), some network issues at our Nevada City location caused our Barracuda e-mail filter to stop responding to inbound message delivery attempts.  This meant that outside messages for Spiral customers were queued for later delivery rather than getting through right away.

The network issue was resolved around noon, at which point there was a large backlog of undelivered mail.  The backlog took about an hour and a half to be processed.  At this point, all undelivered messages from earlier today should have been passed through to Spiral mailboxes correctly.

If you are still having issues with inbound mail not having shown up today, give our office a call at (530) 478-9822, Ext. 1 for customer support.

Posted in Unplanned Outages | Comments Off on Inbound email delays (resolved)

AT&T strike will complicate DSL repairs

According to several media sources, 17,000 AT&T workers have walked off the job today to protest issues affecting technicians — including outsourcing, and being forced to do work outside their area of expertise.

We have confirmed this with AT&T directly, as they are now telling us that “maintenance ETRs have been delayed indefinitely” for newly reported DSL problems.  (Unfortunately, we also cannot guarantee whether or not AT&T will honor commitments for previously scheduled DSL repairs.)

The AT&T strike affects Spiral customers because our DSL service runs on the landline telephone infrastructure owned by AT&T.  When line-quality issues are reported to us, they must be referred to the phone company, who dispatches their own technicians.

The strike follows on the heels of complaints from elected officials statewide that AT&T is abandoning their landline phone network and skimping on maintenance.  California Public Utilities Commission data shows that every month for the past three years, AT&T has failed to meet state telephone quality standards — while 90% of reported outages are supposed to be repaired within 24 hours, AT&T only meets that 24-hour goal about half of the time.  (In Q4 2016, their average was 54%.)

This is one of the many reasons that we are moving forward with our fiber-optic network — where we will own the infrastructure ourselves, and be able to take accountability for its repair and maintenance, rather than relying on a bureaucracy that doesn’t care about the challenges of rural broadband access.  “Zone A” of our network has already received $16 million in grant funding from the CPUC, and we are taking preorders within that zone as we finalize our environmental impact reports and prepare to break ground.  (If you’re outside the zone, go take the survey to let us know where fiber is needed and wanted.)

In the meantime, we will provide the best service that we can to our DSL customers, given the limitations of degrading and unrepaired infrastructure.

Posted in Tips and Announcements | Comments Off on AT&T strike will complicate DSL repairs

Dial-up access issues (resolved)

We are currently investigating an apparent outage with our upstream provider of dial-up services.  Reported symptoms include getting busy signals when calling in to the phone numbers in our dial-up access pool.

Please be patient as we isolate and address the issue.  As of 4 p.m. we do not yet have an ETR.  We will report updates here.

Update, 3/3: Network issues at our upstream provider were addressed last night and tests this morning show connectivity has been restored.

Posted in Unplanned Outages | Comments Off on Dial-up access issues (resolved)

Password compromise on shared hosting server

E-mail, client and FTP passwords on our shared hosting server have been reset due to a suspected compromise of some hosting customers’ account information.

There was another attack this week on our shared web hosting server (see also previous), injecting malicious code into a Spiral-hosted website. This week’s breach, however, was definitively traced to an unauthorized FTP connection using the account owner’s credentials.  Forensic re-review of several of the previous break-ins suggests that that method was employed for previous weeks’ website alterations, and the data available strongly suggests that multiple user passwords have been compromised.

Further review revealed that every single site affected by the recent breaches is one that had been transitioned from our secondary hosting server onto our primary server in mid-2016 when we closed the secondary server down.  The odds of that happening by chance if the breach had occurred on the current server are extremely low.  Additionally, the secondary server had far weaker security than the existing setup (a major reason for the migration).  Both of these facts point to this breach having occurred before the server transition, and old passwords from that breach being withheld for later use.

However, out of an abundance of caution we have reset ALL passwords on the current shared hosting server.

The hosting servers do NOT contain any customer financial or personal information — the only data exposed in a potential breach would be usernames, passwords, website contents and e-mail mailboxes.  However, any sensitive or personal information sent via email to accounts on that server may likewise have been compromised.

If passwords used on that server were reused elsewhere, we advise changing those as well.

Who is not affected

Customers using Spiral for internet access via DSL or dial-up are NOT affected.  Customers pre-registered for our fiber optic project are NOT affected.  Customers with NCCN.net e-mail addresses are NOT affected.

Who is affected

Spiral’s web hosting and custom-domain e-mail hosting customers have data on the server where we performed the password reset.  However, not all of those accounts are suspected to have been breached.

Approximately half of Spiral’s web hosting were formerly located on the secondary server where the data breach is suspected.  If you have a website hosted by Spiral, call our office at (530) 478-9822 and we can review our records and tell you whether you may have been affected.

Customers with spiralEmail.com e-mail addresses, and sierraEmail.com e-mail addresses, have mailboxes on the server where we performed the password reset.  However, those accounts were NOT ever located on the secondary hosting server believed to be compromised; at this time we have no evidence of a data breach on those accounts.

If you have concerns about your data security, or need to speak with us about your passwords, please call our office at (530) 478-9822, Ext. 1 for customer support.

Posted in Unplanned Outages | Comments Off on Password compromise on shared hosting server

DSL outage (cleared 1/31)

As of approximately 1:00 PM Tuesday, January 31, multiple and widespread reports of DSL problems have started coming in. We are investigating this as an area outage and are currently assessing the extent of the problem.

Both Grass Valley and Nevada City DSL customers are known to be affected, with reports suggesting that the outage extends beyond Nevada County. The symptoms of the outage appear to be:

• All lights on the modem indicate a normal connection to the outside world, but no IP layer traffic is getting through.
• Power-cycling your DSL equipment (turning it off and on) does not restore service.

We will update this post as we learn more.

Edit, 1:40 pm: The source of the problem appears to be a major Denial of Service attack against our upstream provider sonic.net.  We are coordinating with them on a response.

Edit: 2:30 pm: An ETA is not yet known.

Edit, 3:45 pm: Our upstream provider is saying that service has been restored, and we are working to verify that across our customer base.

Edit, 4:25 pm: Normal serviceability appears restored.  If your service continues to have problems, please call our office at (530) 478-9822, Ext. 1 for customer support.

Posted in Unplanned Outages | Comments Off on DSL outage (cleared 1/31)

Multiple website compromises / e-mail delivery issues: WordPress attack

In an issue very similar to last week’s outage, our shared hosting server was hit by a compromise which caused high volumes of malicious e-mail messages to be sent out.  Unfortunately, this time multiple customer websites were confirmed as simultaneously compromised.  All of them were using the WordPress content management system (last week’s victim was not), and may have been attacked through the same scripting vulnerability.

As such, we have audited every WordPress-based site on the server, finding a total of seven sites with malicious code additions.  Those websites have been temporarily suspended while we work with their owners to re-secure the sites and remove the malicious code.  We will additionally be contacting every single customer running a WordPress-based site to make certain that all installations have had the most recent security patches applied.

One of the effects of the attack is that large volumes of spam were once again sent out from our shared hosting server, causing it to be placed on e-mail blacklists and watchlists.  Outbound messages are correctly passing through our server, but the reputation issues are causing them to be rejected at the recipient’s end.  This affects customers with e-mail addresses at spiralEmail.com, sierraEmail.com and custom hosted domains.  We have requested removal from every blacklist which allows manual delistings, but restoration of full email delivery may require 24 hours or more for automated listings to expire.

Posted in Unplanned Outages | Comments Off on Multiple website compromises / e-mail delivery issues: WordPress attack

E-mail/webmail login outage for nccn.net customers (resolved)

A problem with the physical hardware running our cloud database server caused user data for nccn.net accounts to be briefly unavailable on Thursday evening, January 19.  This meant that users were unable to log in to their e-mail, either directly via our mail servers or via SquirrelMail at webmail.nccn.net.

(No e-mail was lost, and e-mail delivery continued uninterrupted: only sign-ins were affected.)

Our cloud provider has notified us that the affected server has been migrated to new hardware, and we have verified data integrity and restored e-mail access.  If you are still having problems, please call our office at (530) 478-9822 and we can give you individual support.

Posted in Unplanned Outages | Comments Off on E-mail/webmail login outage for nccn.net customers (resolved)

Outbound email delivery problems

Customers with e-mail addresses at spiralEmail.com, sierraEmail.com and custom hosted domains are experiencing issues today while sending outbound messages.  Some messages are coming back bounced, and some are not reaching their recipient.

The root of the problem is that a website on our shared hosting server was compromised and sent out high volumes of malicious messages.  This caused our shared hosting server to be placed on multiple e-mail blacklists.  Messages are correctly passing through our server but the reputation issues are causing them to be rejected at the recipient’s end.

We are in the process of re-securing the server so we can initiate the process of blacklist removal. It is difficult to give a specific ETA due to the nature of blacklist removal, but the server security audit that allows us to initiate the removal process is almost complete and typically normal mail delivery resumes within 24 hours.

Edit, 12:30 p.m.: The removal request has been submitted to the primary blacklist causing delivery problems, and they estimate a 4-hour propagation window for the changes.  We are reviewing the server’s mail queue to see what other blacklists need to be addressed.

Edit, 4/19: While the blacklist removal was completed within minutes of the request above, caching issues meant that some servers were using the old blacklist information for several hours (in one case up to 24 hours) after the removal was validated.  We have been monitoring over the past two days and at this point it appears that all e-mail traffic is back to normal.

Posted in Unplanned Outages | Comments Off on Outbound email delivery problems

Winter weather

The storms hitting California this month are predictably creating infrastructure challenges — though at a level we haven’t seen for many years.  High winds bring down trees (and power/telephone lines with them), heavy rains soak equipment (causing electrical shorts through poorly maintained weatherproofing), and massive flooding creates mobility and accessibility problems.

We’ve been informed by AT&T that they are in a statewide state of emergency, and are dispatching technicians to address known problems on an as-available basis.  What this means is that if your telephone line is down, or if your DSL service is suffering problems caused by line-related issues, in some cases they are not even providing ETRs (Estimated Time of Repair), or if they are, the repair date is multiple weeks out.  They are also refusing to accept any escalation requests for repairs unless the lack of service creates a medical emergency.

If your DSL service is having problems, you may be in for a  long period of downtime.

Note that if a weather-related outage disrupts BOTH your Spiral DSL AND your AT&T phone line, contact AT&T’s 24-hour support at 1-800-288-2020 and report the telephone issue; in almost all cases, this will result in the fastest possible repair of both services as they address the physical damage.  (Spiral can report DSL issues for dispatch of a DSL technician, but if the damage affects telephone service as well, often they will need to leave and reschedule the repair with a wiring technician.)

If your telephone service is working well and your Internet is down, give our office a call at (530) 478-9822 and we can help you diagnose the issue — DSL equipment can also be sensitive to power outages, and we can help you solve internal equipment problems on a MUCH shorter timeline.

For those of us whose service is still working, the best thing to do is hunker down and wait out the storms, but give us a call if anything needs improvement and we’ll work with you to see what can be done.

Posted in Service Updates, Tips and Announcements | Comments Off on Winter weather

Outbound mail delivery problems (Resolved)

Web hosting customers, and e-mail customers with spiralemail.com and sierraemail.com addresses, are reporting problems with delivery of outbound messages. Many messages are failing to be delivered, with a delivery report being sent back to the sender indicating that they were rejected due to spam violations.

This is due to the hosting server being placed on spam blacklists after a series of accounts were compromised (in unrelated incidents) and malicious messages sent out from each of them. The accounts have been re-secured, but the process of getting removed from the blacklists is taking longer than expected due to multiple violations within quick succession.

We are actively working on restoring full operability, but do not have an ETR at this time.

Edit: Deliverability was restored Sunday evening, July 3.  If you are still having problems please contact our technical support at (530) 478-9822, Ext. 1.

Posted in Unplanned Outages | Comments Off on Outbound mail delivery problems (Resolved)