Outage Update – October 30th, 2017

Update 11/09/2017 10:22pm CST:  As of 10:22 PM CST on November 9th, 2017, all missing data from the KidKare accounting outage has been restored. Please read the following post for more information: https://blog.minutemenu.com/2017/11/10/kidkare-accounting-resolution/

Update 11/08/2017 9:45am CST: The invoices and associated data that were missing from the time period of October 30th, 2017 from 2am – 10am have been restored. We are currently working with our payment vendor to verify that all payment data (including payment status) is accurate. Please continue to monitor this post for more updates.

Update 11/06/2017 5:22pm CST: There are still a small number of invoices and associated payments originating on October 30th, 2017 from 2am – 10am that have not yet been restored. Our team is working on verifying this last section of data before it is fully restored. Tomorrow, November 7th, 2017 we will be working with our payment vendor to verify that all payment data is accurate. Please continue to monitor this post for more updates.

Update 11/06/2017 10:51am CST: Invoices and payment data will be restored over the next few hours. Throughout the day we will be restoring any missing payment information. Please continue to monitor this post for more updates.

Update 11/05/2017 8:16pm CST: Progress with scripts to restore data is still in progress. Our team is working through the night to ensure that the data restored is verified as accurate. We anticipate a final update tomorrow, November 6th, in the early afternoon.

Update 11/03/2017 5:45 CST: Progress with scripts to restore data is currently in progress and testing is going well. Our team is working through the weekend to ensure that the data restored is verified as accurate.

Update 11/2/2017 10:20am CST: The data restoration process for missing KidKare Accounting data has been started. We are verifying the restored data in a test environment today so that we can ensure all data is correctly restored.  This testing will continue through tomorrow so that we can coordinate with our payment processing partner to ensure any payments that were processed are displayed in KidKare and are not duplicated.

We are anticipating having everything restored by the end of the week. If further testing is required tomorrow afternoon, we will continue to restore data over the weekend.

We understand that there may be invoice issues on Friday, November 3rd, and we are prepared to help resolve any issues that our customers are facing.

Update 10/31/2017 10:05am CST: You may still be missing info from last week. This applies to accounting data only such as invoices, payments, and mileage expenses. This does not apply to food program data. You might be seeing invoices marked as unpaid, but they are actually paid; your payments have been processed. This issue does not affect payment processing. No payments were reversed. Our team is still working on restoring your data as quickly as possible.

Update 10/30/2017 4:40pm CST: The process for restoring data from after Friday, October 27th, 2017 is in progress and is scheduled to be completed as soon as possible. Once this process is complete you will have access to all of your data. Please continue to monitor this blog for further updates.

Update 10/30/2017 12:20pm CST: Do not re-enter data that is currently not showing. We are restoring all data from over the weekend. If you re-enter your data you will have duplicate records when the full data restore is complete.

Over the weekend we updated our entire infrastructure. This update was critical for providing a more scalable and reliable environment for customer data. Our data structures, hardware, and physical location of servers were updated during this time. In moving all the data into the new environment we had some anomalies. We want you to know we have not suffered a data loss. All customer data has been retained, the anomalies we are seeing are due to an issue that occurred during the data migration.

Once again, no data has been lost. You may have seen incorrect data, or data that was not visible. This problem is currently being addressed and the complete resolution will happen within approximately 48 hours.

We are currently restoring the data back to Friday evening before the update, which has the most accurate data available. All data you currently see will be correct from that Friday evening state.

Any changes or data entries you have made after Friday will be restored within the next 48 hours. If you have created invoices or added expenses, you will see that data after 48 hours, or once our full data restore has been completed.

We have been receiving a high volume of questions regarding invoices and auto-pay. Auto-pay batches, including recurring invoices, are processed first at 4am, and again at 11am to reconcile any transactions that did not process from the first batch. We have disabled auto pay for today after the 4am batch as a precautionary measure to prevent discrepancies in data entries. This measure should not impact you, but you may temporarily see incorrect data until the full data restore is complete.

Although we had limited information available over the weekend while the outage was being diagnosed, we encourage you to follow us on social media or keep watching our blog for more updates.

Our entire development and support team have been working over the weekend and through the night to get this issue resolved. The support tickets, emails, calls, messages, and comments on social media are not going unheard. We understand your frustration and do not take any outage lightly. We are also working diligently to prevent future outages and improve communication.

We thank you for your patience and hope to have this issue resolved as soon as we possibly can.

Thank you,

The Minute Menu Team

2 Responses to Outage Update – October 30th, 2017

  1. Patty Ahrens says:

    Once I open the blog, how do I print it??

    Patty Ahrens
    Nutrition Works
    1607 W. Jefferson
    Boise, ID 83702
    (208) 336-5533 ext. 6541
    (208) 336-0880 (fax)
    pahrens@jannus.org
    patty@nutritionworks.org

    [cid:image002.png@01D03AED.CA25A7F0]
    http://www.jannus.org

    Nutrition Works is a program of Jannus, formerly Mountain States Group. Mountain States Group is now operating under the new name Jannus. The new name better reflects the group’s commitment to helping individuals and families navigate difficult transitions into more positive futures.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: