Interchangeable Work Area Recovery Exercise (IWARE)

Interchangeable (Mix-and-Match) Work Area Recovery Exercise!

VALUE * VALUE * VALUE

If you have never done a work area recovery exercise, you are in for a treat. If you have had the good fortune of doing them, you already realize why I am so excited about this post.

Let’s examine some of the incredible value you will receive from the IWARE:

• Testing your ability to reach the Incident Commander OR their Alternate with or without letting them know the exact time you will be calling them.
• Testing the validity of the Incident Commander to contact Incident Command, Emergency Operations team members and all the process owners either through an automated call list or a manual call tree.
• Testing the ability of the Incident Command team to meet at a physical or virtual command center to make decisions
• Testing recovery staff knowledge of traveling to the alternate work area recovery location(s), working from home or working from a mobile trailer
• Testing the ability of recovery employees to work productively from the alternate work area recovery location(s)
• Identifying and closing all gaps prior to a disruptive event
• Testing the transfer of customer facing toll free phone numbers to the alternate recovery location(s) (a very cool part of the exercise)
• Insuring you have critical special equipment available at the recovery site. For example, high speed scanners, MICR printers, rubber stamps, form letters…
• Testing the ability to transfer equipment from a third-party storage facility/vendor to the alternate recovery site
• Validating all plans are accurate
• Validating call tree procedures are accurate
• Validating work-from-home recovery employees can log in during the exercise and reach all of their critical systems
• Verifying image configurations and the ability to access production systems from the alternate recovery location(s)
• Building relationships – getting to meet new people, putting a face to a name, making friendships
• Working closely with IT (you may not like it but I ‘grew up’ there, so please be nice…)
• Having a great lunch! That is critical! Don’t cheap out. More below…
• Generating new ideas for improvements from attendees
• Doing a post event meeting (I stopped calling it a post-mortem, as it seemed like a depressing term). Listing each item that needs fixing, assigning responsibilities and deliverable dates and tracking until closure!
• Getting high praise from the people that attended AND your management!!!

Ok, that is a lot of really cool stuff. I ‘affectionately’ call it the exercise that just keeps giving!! To pull this exceptional exercise off coordination, dedication and management support is required. If I was able to do it, you certainly can. You will ace this exercise!

A bit of history:

I have happily completed more than 100 IWARE’s thru the decades for various companies. Just thinking about them brings back such great memories. Some had 150-200 attendees and others had 10 attendees. All were great, valuable, educational and fun!

More importantly, the value clearly demonstrates you can enact your plans and recover to a vendor work area, sister-site, mobile trailer(s) or work-from-home. The IWARE is a key component in preparing and mapping to the actual response AND recovery processes you currently have documented. Yes, there will be minor tweaks at crunch time, but that is where the resilient culture you are building will shine.

Possibly your team is virtual, such as my current team is now and all of my former teams were. The IWARE brought us together during exercises throughout the United States and overseas! There were late dinners (after setting up the network and workstations of course) and time to have fun and rant about many things. I treasure the moments working with such great teammates through the years and how much we achieved.

Value-wise I really like the IWARE because it tests so many different components of your program while never impacting business production. The business will love it as well. You will hear the inevitable excuses not to participate, but once the exercise starts the value will be realized and you will see quite a few smiles.

On one occasion, I did an exercise for 150 people in Massachusetts as the sole representative from our business continuity team. My DR partner was ill and understandably could not attend. It was a real challenge but fortunately, like you, I embrace and enjoy challenges. I had to handle both the DR and Business Continuity facets of the exercise. It was a huge success and a great memory! Although I live in NY, I always eagerly volunteered for the Massachusetts recovery exercises when we created the testing schedule. Boston is my favorite city. I loved going up there and to smaller cities including Lowell, MA.

I have so many memories. I am a devoted basketball fan and once I timed an exercise to coordinate when the New York Knicks were playing the Boston Celtics in the old Boston Garden. I started out from Long Island and about 30 minutes into my trip I was called back to attend to a local disruption. I was so disappointed, but duty called.  I never did get to the old Boston Garden. Ok, not such a great memory. Hey, they can’t all be good. But I suppose my subsequent ‘bucket-list’ trip to iconic Fenway Park to see the Red Sox vs. the Yankees made up for it. Seeing the Green Monster in person was surreal!

Ok, down to business:

The IWARE can be conducted in three ways as a fully announced, partially unannounced or fully unannounced exercise. It can be held over two or more days. You should align the exercise to the BC maturity level of your company:

• Fully announced IWARE: during the preparation phase the process owner and all employees are aware of the start date of the exercise. There are no surprises. If the business has done tabletops to this point and this is their first work area recovery exercise, I suggest conducting the IWARE as a fully announced exercise
• Partially unannounced IWARE: during the preparation phase the process owner and all employees are aware that the start date of the exercise will be during a certain month or quarter. This positions the exercise as more stressful and realistic than the fully unannounced. If the business has done tabletops and one or two fully announced IWARE’s then the partially unannounced is a nice step up in realism
• Fully unannounced IWARE: Surprise!  – the preparation phase is done within your team. The exercise is a complete surprise to the process owner and all employees. This positions the exercise as the most stressful and realistic scenario, short of a real event. I have matured programs to such a high degree that upper management and process owners actually requested we do fully unannounced exercises. It is great preparation, but you have to be at a high maturity level

The off-site recovery phase of the exercise is conducted the same whether the exercise is fully announced, partially unannounced or fully unannounced.

Tip – You will meet success if you have patience and map out a step-by-step project plan so nothing falls through the cracks:

Here is a sample timeline I like to use and the coordination of steps leading up to and thru the exercise. The exercise provides many opportunities to tweak and customize the timeline and scope to precisely fit your needs. Have fun with it:

Prior to the exercise:

• Schedule the exercise the year prior to the actual event. So, if the exercise will be during June 2020, distribute the proposed date when your yearly exercise schedule is published, probably during November or December 2019 (the prior year). This leaves room for the Incident Commander to communicate conflicts. It provides flexibility in case you must adjust the month
• Begin the actual coordination of the exercise approximately two – three months prior to the date of the exercise
• Schedule a webinar overview with the Incident Commander and the Alternate Incident Commander
• Schedule a webinar overview with the process owners. If there are multiple locations in scope you may need multiple meetings, as there will be questions during the webinar and it can get out of hand with too many people. I try to limit these meetings to a max of 40 process owners and their backups, otherwise it can lose the personal touch
• Perform the webinar overview with the Incident Commander and the Alternate Incident Commander. Leave time to answer all questions and concerns
• Perform the webinar overview with the process owners to review the exercise. Leave time to answer all questions and concerns. The Incident Commander and/or Alternate Incident Commander should attend these calls.
• Determine whether the exercise is a ‘go or not’. There may be excuses by local management but upper management should back you up in that all processes must participate
• Determine if there are customer facing toll free phone numbers in scope for re-routing to the alternate recovery location(s). Gather call re-routing requirements. Coordinate with the vendor and your telephony people. (I detail call re-routing in a separate post)
• Coordinate which recovery rooms and seats you will use with the third-party recovery vendor and/or coordinate with the facility director if the exercise will be held at your sister-site(s)
• Coordinate with IT and DR teams to insure systems can be accessed from the recovery site. This will entail a series of meetings up-to and including the day of exercise

Exercise execution:

I am going to describe the exercise as a three-day exercise below. You may get to the maturity level where you can compress it into two days. Day 1 being the notification and day 2 having recovery employees report to the recovery locations as documented in their plans:

Day 1 – Tuesday evening approximately 7:30 pm:

• Place a phone call to the Incident Commander. If you cannot reach him or her call the Alternate Incident Commander. You can also follow-up with an email. Depending on the type of exercise, he or she may or may not be surprised to hear from you
• TIP –I say this in a loving way – Some Incident Commander’s will do anything and everything possible to find out the exact date in advance. They may bribe you, they will check your calendar, they will speak to your friends. They can be verrrry clever. So, use care
• When you call the Incident Commander you will let him or her know the exercise is beginning. They are usually very positive about the exercise. Describe a scenario – fire, flood (something appropriate for the location). Let them know the first step is for them to activate their process owner and crisis teams call lists. They should instruct the process owners to meet at 9:00 am the next morning (Day 2)  to discuss the exercise in a conference room – that you already secretly reserved. Again, be careful when you reserve the conference room. The cunning ones will be looking for that reservation with your name indicated when trying to figure out the ‘surprise’ start date of the exercise. So have an admin schedule it for a fake meeting. As you see, there is a lot of interesting espionage in this exercise
• After you finish your call with the Incident Commander or multiple Commanders if multiple locations are in scope for the exercise, you should send a progress update to management and everyone else who should be in the loop. In fact, progress updates throughout the exercise are critical. Always send them

That is all for Day 1! Easy Peasy!

 Day 2 -Wednesday (three day exercise)

*** if you are running an aggressive 2 day IWARE then Day 3 Recovery would be performed on Day 2. Some of the steps below would then be performed on Day 1 after the phone call to the Incident Commander:

• The emergency operations team meets in the reserved (secret) conference room at the production location and a conference line is opened up for those that have to call in due to travel. It is good to have both
• You or another member of your team should either attend in person or be on the call. You may have to travel to the recovery site on Day 2, so the conference line is important. Also, if multiple sites are in scope for the exercise, you can either stagger the start time of the Day 2 meetings or assign a different BC Team member to attend a different site meeting
• The Incident Commander should open the meeting and briefly discuss the exercise and its goals. The Incident Commander should also discuss the success or gaps in the previous night’s outreach to contact the team. For instance, they may have tried to contact someone and reached a bar. That actually happened to me many years ago during a disruptive event. Fortunately, we had backup contact numbers for that person!  If there were problems reaching people the prior night, these are wonderful opportunities to close gaps before a real disruptive event occurs. I have used that valuable line thousands of times in my career – have you?
• The final take-away from the meeting is each of the process owners have to indicate who will be participating (recovering) on Day 3 (the next day for the 3-day version of the IWARE). The recovery testing should align with what they have stated in their business continuity plan. So, if they stated the primary recovery location is a 3rd party vendor site, then people should test from there. One caveat though, if their primary recovery location is work-from-home, you should encourage them and perhaps make it mandatory that a small percentage of their recovery team test from the 3rd party vendor recovery location in addition to testing from home. This is valuable in case work-from-home is not a viable option at time of disaster.
• Determine the minimum number of staff that must recover during the IWARE. It should be a number that validates true recovery but not high enough to impact production work if there is a problem at the recovery site. I generally require 10%-15% of the normal recovery staff to participate but I always encouraged process owners to send as many people as they would like. The more the merrier!  If their strategy is 20 people would recover during an actual event, a minimum of 2-3 people is required to attend the exercise. You should set a percentage that makes sense for your organization. All processes should be in scope, even if their primary strategy is work from home.
• Remember to send those progress report(s) to management and all stakeholders
• During a 3 day exercise, you and your team will be setting up the recovery environment, turning on the network repeaters, imaging workstations, insuring the attendee list is in place… if it is a 2 day exercise you would have covertly done it on day one or prior to the exercise.
• Make sure if a ‘Chinese Wall’ is required to separate departments for regulatory reasons that you account for it when designing the seating plan.
• Print and place attendees names in front of the workstation they will be assigned to
• Depending on the number of participants you might finish setup at 7pm or 1am
• Grab dinner during setup if it will be a late night or dine at a nice restaurant if you, hopefully, finish setup early
• Get a good night’s sleep. The final day will be busy but so much fun!

Day 3 – Thursday – Recovery:

***If the IWARE is performed as a 2 day (rather than 3 day) more aggressive exercise, recovery would be on day 2.

This part of the exercise remains consistent for fully announced, partially unannounced, fully unannounced, 2 day and 3 day exercises.

• Set 3 wake-up alarms so there is zero chance that you will oversleep. Especially if you and your BC buddies were out late the night before
• Allow plenty of time to get to the recovery location. Have a backup plan if your car does not start
• Buy breakfast for the attendees
• Set up shop at the recovery location with your team. I like to begin final day prep at 6:45 am for an 8 am official attendee start time
• Even if you had set the exercise time for 8 am some people will get there early. Some people may arrive as early as 7:20 am, which is fine. Other people will have trouble finding the vendor site – but that is ok as it prepares them in the event they need to relocate during a real crisis
• Greet participants as they arrive. Escort them to their seats. Thank them for attending and introduce members of the BC team and any IT support people
• Let managers know that they can use private meeting rooms if they need privacy and show them where they are
• Address the recovery associates as a team early when most people have arrived. Thank them for attending and explain where the bathrooms are located, where emergency exits are located, when lunch will be served, etc. Let them know that by raising their hand you or IT will help them with any issues. Ask that they complete a survey before they leave. Make them aware of where they can get the coffee, bagels and donuts you bought earlier in the morning – probably a break-out room
• Recovery associates should work from the provided desktop to validate each of their critical systems and to report any issues and gaps. They should also report any ergonomic concerns. You must compile all of the feedback
• After they have validated their systems some people are more comfortable working from their laptops (if they brought them). I allow them to work from the laptop at that point
• Test and review the inventory of supplies, headsets, high-speed scanners, MICR printers. Replace batteries where necessary. Make sure there are backup batteries and check the expiration dates. Ok I will admit it, I got bit by bad batteries one time.
• If equipment is required from a 3rd party storage vendor, request it be sent over. Keep track of how long it takes to arrive. I have discovered and closed gaps doing this. You will need that equipment during a real event
• Take lunch orders – remember do not ‘cheap out’ on the lunch – it is critical to success
• Serve lunch in a break-out room
• If management permits it employees may be finishing up at 2 pm or 3 pm. It is important for you to circulate a survey mid-day and instruct people that they must complete the survey before they leave while everything is fresh in their minds. The survey can be on paper but I prefer using Survey Monkey or another online tool. Be sure to include the work-from-home people and management in the surveys
• As people leave thank them for participating in this important event.  You will probably get lots of thanks you’s from them as well
• If you tested in a shared recovery space make sure to clear your desktop computer image off the vendor machines and shred any paper artifacts left behind!
• Send equipment back to the proper storage facility or place in on-site lockers
• Perform a hot-wash meeting with the business management team at the end of the day. It is valuable to speak with them the same day, if possible, while everything is fresh in their minds. Discuss what worked, what did not work and what you can do to improve going forward
• Send a report with high level final results. Make sure upper management is included on the report. Again, thank everyone for participating and ‘making this important exercise a success’. Always give credit – where credit is due
• Finally, say goodbye to your teammates in BC, DR and IT until next time

Post exercise meetings

The week following the exercise meet with your BRBC team and DR. Review issues and opportunities for improvement captured during the exercise

• Determine a plan to close all open issues
• Formalize an executive summary and publish it to management and process owners. Remember to thank everyone for their participation in this valuable exercise
• You may want to also do a follow-up meeting with management to review all of the issues in the executive summary
• Remember to indicate in the exercise schedule database that the exercise was completed for each location and process in scope