Amazon S3 Outage: Our Response

What happened?
On February 28, 2017 at 12:37 PM EST, Veoci users started experiencing problems accessing files stored in Veoci as well as uploading new files into Veoci.  The issues were triaged and eventually resolved at approximately 2:30 PM EST.

How did this happen?
One of Amazon’s system administrators accidentally entered an incorrect command which resulted in the complete shutdown of many Amazon servers, thus disabling Amazon S3 services in the US East Region (Virginia) and affecting thousands of customers.  Amazon’s summary can be found here: https://aws.amazon.com/message/41926/

Why this impacted Veoci?
Amazon Simple Storage Service, Amazon S3, is an internet storage system to store and retrieve any amount of data from anywhere on the web.  It is split up among 8 storage locations around the world. Veoci uses Amazon S3 storage in the US East Region in Northern Virginia and the US West Region in Oregon. All Veoci files are stored across several servers in Virginia and are automatically replicated to Oregon. 

What Veoci functionality was impacted?
Most Veoci use was unaffected; however, uploading files, downloading files, importing via Excel, exporting to Excel, emailing attachments to Rooms, and submitting entries into public Forms with attachment fields were inhibited.

How did the Veoci team respond?
Veoci’s technical development team agreed to wait a short time after the outage to see if Amazon would recover quickly. After about 90 minutes without a resolution from Amazon, the Veoci team decided at 2:00 PM EST to modify our configuration so Veoci’s files would be directly accessed and stored using our Oregon location. This failover process was fully in place at 2:30 PM EST. At that time, the only functionality that was still impacted was accessing some previous file versions.

Amazon S3 recovered from the outage at 4:54 PM EST. To ensure that Amazon had fully recovered, Veoci’s failover to Oregon continued for approximately two days, continuing to support almost all normal operations. On March 1, 2017 at 11:50 PM EST, our team reconfigured file access and storage back to the Virginia site, restoring all normal operations. 

How did the Veoci team communicate this issue?
Veoci’s primary tool for delivering outage notifications, Mailchimp, was also impacted by the Amazon S3 outage. After exploring other communication options, we emailed all Organization Administrators at 2:03PM EST.  An update was also sent to the same audience at 2:52 PM EST.

Next Steps
To further improve our recovery method, we are currently working to configure all Veoci files to be automatically stored and accessed via both the East and West Amazon S3 Buckets. This way, should the Virginia location encounter another issue, files will automatically be sent to Oregon with no failover process needed.

Veoci is also implementing a capability for our team to broadcast an Incident Alert to all users directly within Veoci, removing our reliance on Mailchimp for distributing timely communications.

A Virtual Approach to Creating a Proactive Emergency Management Program for Airports

A recent issue of Airport Improvement Magazine featured an article by John Kinney, chair of the American Association of Airport Executives (AAAE) Emergency Management Committee, that highlights the importance of airports to define their own Emergency Management Program (EMP). Airports vary drastically in size, operational capacity, and functionality and therefore need flexibility in developing an Emergency Management Program to meet their needs.

Part 139 and TSR 1544 regulations do not capture all the needs of airports to create a comprehensive plan for when an emergency occurs, despite the consequences of having an inadequate EMP. An effective EMP, according to Kinney, must reach all corners of the airport environment–stretching far beyond what the regulations mandate. For this reason, Kinney suggests that an effective EMP pulls not only the first-responders, but communities on and off the airport into the conversation as well. Airports should establish an EMP that follows guidelines across multiple regulations, while incorporating those factors which are unique to the specific airport’s environment exceeding federal guidance.

Considering the above, for Kinney, effective airport emergency management “entails an empowered, proactive strategic group of planners and facilitators who organize resources, personnel, technology, facilities, business and community assets and capital through an organization structure of ICS to deliver the greatest level of resiliency of an airport, expediting the recovery from irregular events.”

A number of Veoci’s processes can help establish Kinney’s vision of a comprehensive, adaptable, and organized EMP that fosters resilience and proactivity.

User End Customizability

Veoci’s point-and-click customization reflects Kinney’s view that an EMP should address factors of the airport’s unique environment. Veoci’s easy-to-use interface allows administrators to build forms, design plans, define gradation among users, and develop escalations all through a drag-and-drop or point-and-click configuration. Without any IT or coding experience, administrators can custom-tailor their digital EMP to fit the specific needs of their airport. In addition, administrators don’t have to wait for a new release or call IT support to reconfigure the EMP to change or modify a plan.

Real Time Communication

“Real-time communication, or as close to it as possible, is necessary to reach every corner of your airport and even off-airport facilities impacted by an event.” (Kinney, 2016,p. 22)

Kinney identifies real-time communication during an emergency as critical for a successful EMP. Veoci is built around an internal real-time communication engine. With a private chat feature for individual users, or a virtual room for groups communication, Veoci prepares airport staff to be in constant communication during an emergency.

screen-shot-2016-12-13-at-4-48-14-pmAn effective EMP encompasses every aspect of the airport organization. It is impossible to have every member of these organizations, or even a representative from each, in a physical room especially during a time-sensitive emergency. Veoci supports seamless and structured communication through virtual rooms, or workspaces, where users can communicate across departments to share time-critical information without being constrained to a physical space. Partitioning your airport departments in side-rooms has the added benefits of security and organization. Within these virtual rooms or workspaces members of every department can upload relevant images or videos, track the progress of a plan, or communicate interdepartmentally.screen-shot-2016-12-13-at-4-51-29-pm

Veoci empowers executives or stakeholders who may not even be at the airport during the time of the emergency to be completely up to date and track to progress of the response as it happens in real time.

Organizational Structure

Non-first responders, who will otherwise be left in a ‘gray area’, need clear roles during an emergency. Airport employees who are non-first responders are often left out of the EMP picture, which can add to the disorganization and pandemonium of an emergency. With Veoci, anyone with a smartphone and a login to the system can be given real-time information about an emergency. Pre-made customizable plans can be launched at the click of a button to prompt users to follow a checklist–clarifying and directing everyone’s role in emergency response.

screen-shot-2016-12-13-at-4-45-41-pmVeoci also empowers administrators to assign roles to groups of employees, and customize extensive plans for all kinds of emergencies before they happen. For example, in the event of an aircraft accident, administrators can push email, text SMS, or phone calls to anyone included in the airport emergency plan–including those who may be off-airport.

Veoci can help airport managers and executives transform their emergency management program from reactive to proactive. Through predetermined forms, plans, workflows, virtual rooms, and an internal real-time chat, your airport can be innovator in the EMP field.

2016: A Year of Rapid Growth for Veoci with No Sign of Stopping

Say what you will about 2016, but here at Veoci, we are celebrating a year of growth and success. We have surpassed 100 customers and we closed the year with a 100% growth YOY. As our customer base expands, so does our diverse team as we gained along with new people, new and unique expertise to aid in the development of new and exciting use cases. Our attendance at Trade Shows has been particularly exhilarating – the crowds at our booths are testimony to the BEST solution for your complete Emergency Management needs – from planning, triage, plan execution, and finally review and improvement.

Expanding past Emergency Management, our users are discovering in Veoci the winning concept of a fixed-price SaaS platform with which an organization can replace tens of applications while also digitizing many processes that still used Excel spreadsheets or one-of standalone applications. Under the hood, Veoci is a powerful no-code application development platform with integrated communications developed with know how from tens of thousands of processes across just about every business domain.

Our diversity is another source of pride for Veoci and the State of Connecticut recognized that, granting us a Connecticut Diversity Certificate. Our New Haven office is bustling with excitement as another year is sure to bring more opportunities for growth and positive change for both our team and organizations who are looking to make 2017 a more safe and proactive year.
 

Welcome New Customers:

Veoci is proud to team up with the many organization that came on board in 2016. 2016 has been a year of travel, shaking hands, building relationships and pioneering amazing solutions and with each new sale, the team feels a new burst of motivation to keep going and keep growing.

Here are a few of our 2016 wins:

 

Aviation

  • DFW: Dallas/Fort Worth International Airport – We are thrilled to have the 4th largest airport in the country, and the 10th largest in the world, as our customer. DFW uses Veoci to conduct Part 139 and daily inspections, manage, map and query work
    orders, prepare for FAA inspections, create checklists of duties for each staff member, create automatic reminders for staff members, map wildlife on our interactive maps, conduct wildlife reports, and push them to the National Wildlife Database, and integrate with the Digital NOTAM System to access logged NOTAM’s.
  • PDX: Port of Portland – The Port of Portland organization is responsible for overseeing the #30 busiest Portland International Airport, and Hillsboro and Troutdale Airports. Veoci will help the three airports digitize their Part 139 Operations inspections, Part 139 work order management, wildlife management, operations and daily activity logs, dashboards to organize all information and integration with a number of other systems for weather and flight tracking.

 

Government:

  • City of Des Plaines – After Lake County, our first municipal customer in the Chicago metro area. Veoci is well underway in breaking into major cities, but with our Regional Director for the Midwest being a Chicago native, and a diehard Cubs fan, this deal was a home run.

 

Corporations

  • Celanese Corporation – Celanese is a Fortune 500 chemical and materials company with 35 plants around the world. Celanese’s Environmental, Health and Safety group purchased Veoci for incident management coordination and communication for their headquarters in Irving, Texas, as well as their international manufacturing facilities, and for notification of the company’s 7,500 global employees.

 

Utility

  • MidAmerican Energy – MidAmerican Energy, a Berkshire Hathaway company, based in Des Moines, Iowa, services almost two-thirds of Iowa, as well as portions of Illinois, South Dakota, and Nebraska. MidAmerican will use Veoci to manage emergencies and major incidents and run drills in preparation for emergencies.

 

Higher Education

  • University of Central Florida – What’s even cooler than UCF signing with Veoci is that they jumped into action and began using Veoci to respond to Hurricane Matthew, just days after purchasing. UCF utilized Veoci for real-time messaging, tracking of campus statuses, and information sharing as they kept people up to date on weather forecasts while pulling in NOAA prediction data into Veoci. 
  • Claremont Consortium – Claremont Colleges is a consortium of five undergraduate – Scripps, Pomona, Pitzer, Harvey Mudd, and Claremont McKenna –  and two graduate institutions, Claremont Graduate University and Keck Graduate Institute. They will use Veoci to unify and facilitate communication between all 7 campuses by logging and mapping events and incidents, creating automatic reminders and updates, managing crucial assets, creating customized contact lists for check-in calls and storing all information in one central database.

 

Software Enhancements

Our team began Agile development 15 years ago at a time when Agile was not even a word to describe rapid software development. The two-week production release cycle is deeply embedded in our culture. Clearly, just releasing a new version every two weeks is not success – it is doing so to meet customer needs, manage security, keep up quality with serious regression testing, making sure performance never suffers and MOST IMPORTANTLY, always backward compatible. (“Can’t be breaking old stuff while adding new stuff”, as they say).

All Development is for naught unless the system is secure and safe from malicious attacks. For Veoci, Security is always #1. Security gets 100% attention. Hosted by Amazon Web Services in the Cloud, and architected with TOTAL encryption end to end, Veoci offers a level of security difficult for standalone on-premise software to match. We are gratified that a Qualys SSL test grades us at A+.

 

Working with our Customers:

We derive value when our customers use the new features and functions. Of course it helps that the features they are seeing may have been requested often as little as a week ago!
Our solutions team works closely with our customers for implementation and continuous improvement. With a two-week cycle of new features and functions, the challenge is to keep our customers informed and to help them take advantage of these improvements. With regular seminars and emails, our team works hard to keep our customers updated. Our customers also pick up on our continuous software development process and do the same with how they use VEOCI.

 

For 2016 we made a extra attempt in a few areas of VEOCI:

> Mobile App – Mobile app usage continues to increase and is now 50% of our usage

> Navigation – What shows up on an automatically personalized masthead

> User Interface – We now have 100% colorblind-friendly palettes

> Communication – Exciting. Call us. Notify one, notify ten thousand. Conference calls.

> Google Maps – Google Maps continues to astound us with improvements

> Dashboards – Veoci was built for real-time view of the world and we have an incredibly exciting improvement that we are in the process of filing a patent on.

> Workflows – Our experience with workflow automation extends to tens of thousands of work processes. We have some improvement in workflows every two weeks.

> The Cardwall – Show your assets, resources and rearrange them for time slots or other variables just as you would by manually moving post-it notes on a wall.

> Integrations with external systems – Many of our customers have single sign on systems and we offer an easy and rapid integration (our secret sauce: our team has been developing authentication methods for fifteen years).

 

Events

We continue to increase our participation in major events in our field, not only to exhibit our product to potential customers but to meet and spend time with current customers.

 

Trade Shows – From our first time at the International Air Safety Summit in Dubai, to revisiting some favorites like the American Association of Airport Executives (AAAE) in Houston and the International Association of Emergency Managers (IAEM) in Savannah, feedback relayed back to New Haven was nothing but positive. One word: Gratifying.

 

Urban Shield – Veoci is proud to have participated in an event as valuable as Urban Shield. Urban Shield is a 48-hour, planned training exercise that began in 2007 for regional fire departments, medical services and local, state, federal and international first responder and law enforcement teams to prepare for and provide a unified response to disasters. Four medical checkpoints were assembled, fully equipped with volunteers, doctors, nurses, and all necessary medical supplies. The Veoci team trained volunteers on how to enter heart rate, breathing, blood pressure and other medical information into Veoci where volunteers at the next checkpoint can access it by simply scanning the patient’s badge, making it better prepared for the arrival of the next team. Click here to read more about our participation in Urban Shield.

 

2016: A Busy Year

As you can see, we’ve had a busy year. With each new sale, renewal, product enhancement or event we attend, we continue to improve VEOCI, ensuring that it is the best tool to aid airports, airlines, universities, hospitals, healthcare systems, enterprises and governments in their preparation for, response to and recovery from emergencies and for daily use in managing operations and incidents.

Missing from photo: Naina Rana, Mike Heaton, Vincent Jessel, Shaleah Williams, Chance Jackson, Molly Zeff, Kaustav Mitra, DaLee Kicker, Matt Wilson, Ray Konareski, Alex Nguyen, Richard Smith 

Epidemic Management: Responding to Zika, or any Disease, in a Virtual EOC

screen-shot-2016-11-18-at-1-42-07-pmWith over 4,000 cases of Zika in the U.S alone, Zika is no longer a problem only for those who have visited South America recently. As the threat has become more and more pressing, hospitals and public health systems across the country are receiving grants and looking for the most efficient and effective way to manage Zika.

A similar process was taking place back in the spring of 2015 as Ebola outbreaks were spreading and shaking up the health care system. Yale New Haven Health System teamed up with Veoci to create a “Playbook for Disasters” as a real-time solution to manage Ebola preparedness, drills and responses to suspected outbreaks. Click here to read about our Ebola playbook with YNHHS-CEPDR.

The features that make the playbook successful are features that Veoci is proud to share with hospitals for just about any type of crisis or disease outbreak. Many hospitals and public health systems are turning to a virtual EOC to streamline their preparedness and response processes, provide full visibility and centralize all communication and data on one, single platform.

Bringing Everyone onto the Central Information Hub

Public health officials are faced with the task of coordinating information with local, state, federal and even international organizations that are tracking the same disease. Healthcare providers, hospitals, other local/county agencies and private sector organizations are geographically scattered, but are instantly united in a virtual EOC in times where immediate, real-time communication is key.

  • Share messages, pictures, videos, files, and locations instantly in a real-time chat room and integrate with your emails
  • Send alerts and notifications across all departments for reminders, urgent messages or check-in requests
  • Create custom forms on a drag-and-drop template and automatically generate reports to collect relevant data
  • View all high-level information from a customizable dashboard
  • Stay connected and access all features from anywhere via our mobile app for iOS and Android

Activating a Plan: Preparing for Zika

Prior to a case of Zika or any major disease ever appearing in your county, health care officials need to have a preparedness plan that is tailored to their unique processes.

  • At the click of a button, assign tasks and workflows as you track progress each step of the way to completion
  • Instantly access and review automatically generated reports and dashboard information for quick decision making

Mapping and Tracking Data Visually

In order for a plan to activate and unfold smoothly, key people, assets and resources must be ready and in place.

  • Enter preexisting demographic data and map people, resources and assets as well as their condition and current status with color-coded icons on a digital, interactive GIS map
  • Share map access with the public and media to ensure valuable information is shared with those who need to know

Security Measures to Protect Data

With critical and sensitive data, security is everything.

  • Ensure HIPAA compliance and industrial grade security
  • Compartmented information with different access levels to ensure certain information is only accessible by those authorized

One Stop Shop

Maintaining an up-to-date one stop shop for all communication, data collection, reports, your paperless paper trail of customized forms related to Zika, all maps showing assets and demographics data, and, of course, your means of instant notification during an emergency, ensures that the response team feels more prepared to respond collaboratively across departments and sectors. And after the response is over, you now have a full record of every action and exchange that took place. For those who want a streamlined, efficient response to any crisis or outbreak that may arise, paper is out and virtual communication and response centers are in.

Interoperability: How Universities, Towns and Counties can Plan and Respond as One

jmu-screen-shot As another winter storm season quickly approaches (we had our first snowfall here in New Haven last week), our customers are preparing their responses. Although our many customers in Southern California are not too worried, our east-coasters have a lot to do to ensure that resources are ready and that their whole region is prepared for how it may be affected. Many of our university and municipality customers take advantage of the ability to connect their EOC to the city and surrounding county EOCs for effective responses to emergencies or severe weather as well as the planning and management of large-scale events.

In 2012, the City of Harrisonburg, Virginia and Rockingham County conducted a joint emergency response exercise for their public safety, police, fire and other departments. Their joint departments respond to over 110,000 emergency calls every year. Their main conclusion? They needed to come up with a better communication and documentation system to link all of the various agencies responding to emergencies or events in an organized, seamless way.

After receiving a grant from the state for a web-based EOC, the City of Harrisonburg began their research with budget and customization in mind. They found product after product with cookie-cutter templates that allowed no freedom for modification and with no ability to link to the Virginia EOC. Harrisonburg understood that Virginia Tech and the University of Virginia, the City of Charlottesville and Albemarle County were planning on purchasing Veoci within the next few months of 2014. After seeing UVA’s planned coordination with Charlottesville and Albemarle County as a similarity to what Harrisonburg hoped to implement with James Madison University and Rockingham County, Harrisonburg jumped ahead and purchased Veoci.

First Things First: Communication

The City of Harrisonburg shares the same values as the Veoci team, that communication is key. They immediately made use of Veoci’s instant chat feature to share weather and traffic updates across departments while Harrisonburg’s PIO used these updates to share information with the public. They created side rooms for heavy storms and major campus events, such as JMU’s Spring Fest, an event for which they used Veoci to aid them in managing and ensuring the safety of 8,000 students partying across 4 city blocks. Pictures of the masses were shared instantly through Veoci’s chat so police, other agencies and their incident commanders knew exactly what was going on and where.

Mapping Graduation and Football Games

After seeing a presentation by UVA on how they use Veoci to manage their football games, Harrisonburg and JMU decided their next step was to come up with a plan to manage their own graduation ceremonies and football games. To make the current status of the various venues for these events easily accessible and clear, they plot color-coded points with coordinating forms to document and show location, status and other information for each venue. Using the same feature, they plot the location of JMU Campus Police, Virginia State Police, Rockingham County Sheriff’s Office and the Harrisonburg Police as well as the exact site of EM teams, ambulances, fire trucks and treatment areas. They also highlight roadblocks, traffic redirections and indicate open or full parking lots.

See Veoci in Action

Coordinating a region’s university, city and county has proven to be an extremely efficient and effective way for Harrisonburg, JMU and Rockingham County to provide unified plans and responses to any situation that may arise. To find out how the University of Virginia does the same in response to winter storms, join us for our webinar on Thursday, November 17, 2016 and 1:30 PM EST. Lucian Mirra, UVA’s Emergency Management Coordinator, will share how they stay in communication, map road closings and keep resources in check on one single, digitized system. Click here to register now.

Veoci: A Buzzword at IAEM 2016 in Savannah

screen-shot-2016-10-28-at-2-37-05-pmLast week, three Veoci team members spent three days in what they say is one of the last cities of the real Old South, Savannah, Georgia. The International Association of Emergency Managers Conference & EMEX brought over 1,000 emergency managers, public safety officers and many others to the charming city. For Veoci, this was our fourth year exhibiting at IAEM and we have been able to witness our growth year by year.

From getting to finally meet some new customers face to face or hanging out with familiar, long-time Veoci users and friends, it is always an exciting atmosphere at our booth. Some of our happy customers even volunteered to share their experiences with Veoci with some potential prospects. From Andrew Marinik, Assistant Director of Emergency Management at Virginia Tech to Rick Fontana, Emergency Manager of our home city, New Haven, Connecticut, our booth was buzzing with discussion of best practices and new use cases.

screen-shot-2016-10-28-at-2-41-00-pmA key difference that our team members notice each year is the decreasing number of people to whom we have to explain the concept of Veoci. After 5 years of creating a name for our company, we find “Veoci” steadily becoming a buzzword that almost everyone has heard of. From organizations looking for an alternative to their current system to organizations who still do everything on paper, our booth was constantly busy and our Solutions Director was demonstrating our software nonstop, essentially, for two and a half days straight.

As our team and solutions continue to grow, we find more and more opportunities to share our software with organizations who are scrambling for an streamlined way to manage all the aspects and teams involved in emergencies or day to day tasks. Meeting people, listening to their stories, and adapting our software to fit their needs is what Veoci does best and after another successful year at IAEM, we continue to do just that.

Veoci’s Rapidly Growing Aviation Solution and Team

screen-shot-2016-10-14-at-2-20-46-pm

Airport Improvement Magazine recently published an article about Nantucket Memorial Airport’s transition from paper to our cloud-based platform, Veoci. Nantucket Memorial Airport signed on with us in 2015 and uses our system for a wide range of processes—from fueling aircrafts all the way to processing employee vacation time.

The article underscores a larger pattern of airports switching over to computer-based systems for managing their Part 139 operations, airfield assets, properties, and emergency preparation and response methods. Veoci employees have noticed the increase in demand as the number of participants in our airport webinars has really taken off, along with the number of responses to requests for information. What seems to attract many airport managers, executives, and operations staff to Veoci is the customizability and flexibility of the platform. Due to the recent increase in customers in the aviation sector, we have added two new employees who embody the flexibility and versatility of the platform.

Our newest Solution Engineer, Alex Nguyen, comes from Lakeland Regional Airport where he led the implementation of Veoci as an Operations Specialist. Alex graduated from Florida Institute of Technology where he studied Aviation Operations and received his Commercial Multi-Engine Pilot certificate. He also received a Masters of Business Administration from Florida Southern College. Alex displayed his creativity and inventiveness by operationalizing and managing Lakeland Regional Airport’s mowing schedules using Veoci, another solution that Airport Improvement Magazine has featured. Veoci for mowing management processes has since become a best practice for other airports to follow suit. Alex’s multifaceted experience in the intersection of business, aviation, and technology is exactly what our platform is all about!

Our newest aviation Account Executive, Vinny Jessel, also brings an interdisciplinary set of skills to the team. Vinny is a recent graduate of Skidmore college where he studied Psychology and the Liberal Arts. He worked as a Research Assistant for a clinical and developmental psychologist for number of years gaining valuable quantitative analytical skills, while refining his writing and interpersonal skills. Vinny also attended Jacksonville University where he studied Aviation Operations and received a private pilot’s license and instrument rating.

Vinny and Alex hope meet Veoci customer’s needs and bring forward new and innovative ideas to help airports function safely and smoothly.