Disaster Recovery in Fiber Optic Networks

In the past decade our globe has been battered with one disaster after an­other. There have been hurricanes, floods, ice storms, fires, earthquakes and volcanoes. The list seems endless. Unhappily, climate change is likely to worsen this situation as the 21^st cen­tury progresses, which will cause enor­mous problems for companies deploying modern fiber-based communica­tion infrastructure.

Climate change models advise us to expect more precipitation for the north­ern United States and more droughts in the Southwest. Heat waves will become more intense; once-in-20-year extreme heat events are projected to occur ev­ery two or three years. The intensity of hurricanes is projected to increase while their speeds will slow, exacerbating the damage they cause. Global sea levels will rise another 1 to 4 feet by 2100, increas­ing urban flooding around the globe.

In terms of preparation and recov­ery, it is less useful to categorize by type of catastrophe—flood, tornado, fire, earthquake—and more useful to look at their predictability. Unpredictable di­sasters can be devastating, whether it be a magnitude-9.0 earthquake off the coast of Japan that produces a huge tsu­nami, or a tornado in the U.S. Midwest. Dams break, bridges collapse and wild­fires burn, often resulting in subse­quent mudslides.

In contrast, predictable disasters can be statistically analyzed and forecasted. We know there will be more blizzards and ice storms. There will be another hurricane with both flooding and wind-driven network damage. With predict­able disaster preparation and recovery procedures, including those for rebuild­ing fiber-based communications net­works can be developed and practiced well in advance.

The good news is that passive fiber is usually rugged and damage-resistant. In some cases, it can even be submerged and as long as the endfaces remain mated, the damage can be minimal. Of course, each disaster will require a dif­ferent response to repair, restore and re­furnish the fiber networks upon which we all rely. Let’s take a look at the pro­cedures and tools we can deploy to keep our networks running.

Prevention or planning?

In terms of prevention, there’s not much you can do. If the cable close tor com­munications rack is in the basement, it’s going to flood. But the best disas­ter recovery plans are established and rehearsed long before the disaster hits. That means thinking about the possibil­ities, and pre-positioning supplies to be deployed easily.

It is essential that repair techs be provided the proper tools and the proper training in advance of the event.

If a tech fails to effectively in­spect and clean during the initial re­pair, it could cause system failure, ulti­mately resulting in more time and costs to rework the repair for a second time. Better cleaning up-front is like a good insurance policy, which minimizes the cost of a disaster.

Access procedures

During fiber network disaster recovery, the first challenge is access. Avoid downed power lines and flowing flood waters. If water cannot be avoided, waist-high waders are crucial tools. In addition to the obvious risks of being swept away, flood waters are nasty and polluted. They carry sewage, disease, live animals including snakes and fire ants, dead animals, and a wicked brew of household and industrial chemicals.

Once at the work site, it is highly recommended that every tech first makes sure everything is safe. Follow OSHA (Occupational Safety and Health Administration) safety standards for all procedures. The building, tower or man­hole should be structurally sound, free of vermin and more-or-less dry. Electrical systems should be powered off. Backup batteries should be disconnected. Be cautious for electrocution hazards, toxic hazards, and leaking fuels (like improp­erly stored gasoline or damaged contain­ers of alcohol). Don’t expect the lights to be working, so us an LED flashlight, which provides strong, reliable lighting without sparks.

Another worry is air quality. Outside-plant cables are made of rubber and polyethylene, which is very toxic when burned, so good ventilation is essen­tial. Never enter underground or en­closed spaces without ventilation and supervision. Follow OSHA “enclosed space” rules with which everyone should be familiar.

Repair procedures

You can expect the facility to be a mess. Start with presaturated alcohol wipes to remove grime, soot and debris. These wipes have an added benefit of disinfect­ing surfaces. In addition to cleaning the fiber, the wipes can be used for clean­ing active components. Nonflammable, high-pressure contact cleaner aero­sols are very useful to rinse particulate from hard-to-reach places. You might be able to clean the surviving hardware, workbenches, keyboards and racks with optical-grade nonflammable dusters.

Now, as to the fiber itself, assess the damage first. Use your test equipment to troubleshoot links. Remember not to look directly into the connectors, as hu­man eyes can’t see the light produced by fiber-optic transducers and the risk of eye damage is real and permanent.

There will be no shortcut to the res­toration process. Every endface needs to be opened, tested and inspected. Don’t assume mated cables and patch cords are bad; fiber is surprisingly resilient to flooding because the physical contact keeps water and debris out of the central contact zone. Nonetheless, bring replacements and be ready to test them and clean them.

Mechanical “push-to-clean” tools generally won’t get the job done. While they’re handy for quick-and-dirty cleaning of lightly contaminated end-faces, you’ll need sturdy cleaning sticks for the more thorough and rigor­ous cleaning required in this environ­ment. Quality cleaning sticks are help­ful in two ways: they clean the entire endface (not just the contact zone), and they are able to clean the sides of the adapter sleeves.

Be careful as to which cleaning sticks your company selects. Some compa­nies try to use cotton swabs, but they will fray on the sharp vertical edges of the adapter sleeve and leave de­bris entrapped in the adapter. Another weak choice is an inexpensive retic­ulated foam swab. By its very nature, foam is a dirty cleaning material and is doubly cursed because it generates static electricity. The sharp edges of the adapter will shave chunks off foam swabs and deposit them on the endface, while the static will cause the particu­late to stick to the endface.

In our experience, wet-dry cleaning works best on endfaces. A fast-drying solvent will help dislodge contamination and remove oily residues. Slow-drying fluids, such as water or even the popu­lar alcohol cleaners, are sub-optimal and will slow the repair. Be sure to select a cleaner in a hermetically sealed, leak­proof container. Refillable pump-bottles will not keep the solvent clean enough for proper repairs and may just spread contamination around.

Lastly, when trying to restore a net­work damaged in a disaster, it is abso­lutely essential techs are provided with high-quality, optical-grade lint-free wipes. These will be used to clean jump­ers and can remove soot, grease, silt, and particulate from endfaces. Quality wipes can serve double-duty and be used to wipe cabinets, racks, workbenches and other surfaces on the first try, speeding the restoration process and simplify­ing inventories.

Disasters, whether predicted or from out of the blue, will continue to wreak havoc on fiber networking systems. The best approach to manage them is to be prepared. Do some preplanning and have a good solid recovery and repair strategy in place. Be proactive by assem­bling the recovery tools needed and pro­viding repair techs with the knowledge and training to make fiber disaster repairs both safe and successful.

Disaster recovery tools for fiber network

What tools should a repair tech have? A good list might include the following:

• Portable lighting with extra batteries
• Replacement patch cords
• A low-power inspection scope, for inspecting the connector endfaces. Be sure to include adapters for the most popular configurations of ports and jumpers
• Hand tools like strippers, cleav­ers, gloves, pliers and screwdrivers, because anything prepositioned will be lost, destroyed or covered in grime. Special tip: Avoid cheap tools because this is a time-sensi­tive situation, so don’t cut corners.
• Assume the labels on the fiber bundles will be de­stroyed or unreadable. Bring sticky-stuff remover and a la­bel-making machine.
• Visual fault locator and continu­ity tester, because if the fiber racks have been toppled or damaged the fibers may have been actually bro­ken or crushed.
• Presaturated alcohol wipes in rug­ged plastic containers
• Nonflammable, optical-grade dusters to blow dry debris from surfaces
• Nonflammable contact cleaner sprays, to flush dry debris from surfaces
• Nonflammable, fast-dry­ing, optical-grade fiber-optic cleaning fluids
• Painter’s tape, to pick up glass shards
• An optical time-domain reflectometer is essential to isolate where problems are
• Optical-quality lint-free wipes, not the cheap ones packaged in a cardboard box
• Battery-powered light source for troubleshooting
• Drinking water
• All of this equipment should be in a hardened case suitable for harsh environment