🚨 Data Store issues: Update record erases fields that were left empty

[Taras_Molotilin]

Hi, we’ve encountered a troubling behaviour in Data Store modules, I’ll try to explain the issue.

For 2+ years we had a scenario implementing a telegram bot, where we’d used Data Stores to track the bot’s internal state for each user (status, telegram id, last message id etc). In this pipeline there is a sequence of Data Store modules: Search, Update, Get.

User gets searched by telegram id, and then the record might get updated. The intended behaviour is that the Update module only affects the fields it is provided with. Those left empty should not be affected. And I want to stress that it did work like this all this time, we haven’t changed anything.

Here’s an example of a run (yesterday, 20:17 our local time), with the configuration of the module:

and the result, after we run a Get Record (sorry for the crop, but it’s still visible):

So, everything that is not set in the Update Record, is kept as it was.

Here is the same scenario, with no changes, 50 minutes later (21:06 local time):

As you can see only two fields are left, specifically, those, that were set in the Update record. The other ones were just plainly erased.

I hope it’s a minor technical issue and it won’t affect your scenarios en masse, but in case you’ve got any pipelines including Update Record (chances are you might :)), please, make sure they are running as intended.

Have a nice day, everyone!

[Michaela]

Hello @Taras_Molotilin

Thank you very much for raising this issue here in the community

Our dev team is aware of this and is now working to investigate and implement a solution as soon as possible.

I will keep a close eye on this and make sure to let you know in this thread as soon as I learn anything new. However, if you’d prefer to receive updates directly in your inbox, please open a ticket to ensure you receive automated notifications.

I am very sorry for any trouble this is causing. Thank you very much for your patience and understanding.

Edit: You can now follow our Status Page for updates.

[null]

Just have to add that this issue is KILLING us. Obviously, datastores, datastore records, and the modules that support them are a core part of the integromat / Make offering, and for them to break like this, resulting not only in scenario failures but in large scale data loss is extremely off putting.

I’ve spent the last 15 hours dealing with the issues on our own scenarios. It seems that opening each “update a record” module, clicking into every field that does not contain an explicitly-defined update value, deleting / backspace, then saving the “changes” to the modules and the corresponding scenarios causes the updates to no longer delete all the rest of the data from the updated records, but it’s a massive PITA, and still leaves the problem of restoring all the data that’s been lost.

The issue appears to have started for us at 10:30AM PDT, Sept 18, and I submitted a ticket within 30 minutes. I got a response 12 hours later, and in the following conversation, the recommendation was to “refrain from using scenarios associated with the datastore app” until the matter is resolved.

This is not possible for us (and I’m sure for others), as we may as well shut everything down at that point.

I write in the hopes that Make understands customers can’t just stop using and updating datastores, and my fear is that after spending the day working around these issues, new ones will arise when Make pushes a “fix” for them.

Very scared to go to bed and see what’s broken in the morning…

[Michaela]

Hello again @here

Just wanted to let you know that the team has identified the issue and has deployed a hotfix which should prevent this from happening from now on.

As mentioned before, I’ll make sure to share any more info or updates that I learn about this.

I understand that this is causing a lot of frustration and for that, I would like to genuinely apologize on behalf of the Make Team.

Your patience and understanding are greatly appreciated

[Michaela]

@here

Just sharing the latest update as also posted on our Status Page:

We have identified an issue with the “Update a Record” module in the Data store app. In the period of September 18th 7:16pm CET until September 19th 12:00pm, the module might have produced incorrect output and data store records that were updated during this period may have been incorrectly stored (only the updated values for a given record were stored while other values were cleared). We are investigating the scope of impact and will provide more information in the next update.

[alex.newpath]

We lost data in several customers data stores as a result of this — configuration data that took many hours to collect. Will there be a way to restore the lost data records in the affected data stores?

[Michaela]

Hello @alex.newpath, thank you very much for raising this question.

The Data Store is used in many different ways, so rather than apply global changes to everyone’s Data Store, we’re currently developing a tool that will allow you to choose the best fix for you. This means you will need to complete a review process to fully rebuild your Data Store.

We expect an initial version of this tool to be available for you by the end of the day (22 September). It will allow you to download JSON documents with both backup and live data, so you can review and fix the data manually.

In the second phase, we’ll provide a simple and comprehensive visual tool. We will let you know when both of these tools become available.

If you need to correct your Data Store before this tool is available, there is a manual workaround. You can review execution logs to locate where items were updated in the Data Store. You can then update this data manually. You can find out more information about the workaround here.

Our systems are now working normally, but we want you to know that we’re here to help put things right. If you have any questions, our support team is here to help. You can log a ticket with them here.

We are truly and deeply sorry for the disruption and additional work this will cause you.

[alex.newpath]

Thanks for the update. If you could post access to the tool to retrieve backup data from the data store that would be great.

[Michaela]

Thanks for the additional question, Alex.

We are currently in the final stages of development, and we will be sharing access to the tool with all affected users via email later today.

Once again, thank you so much for your patience.