The hotfix seems to do its job. But I have to watch this over a longer run. While watching the material more closely I discovered that there were occurrences much longer ago in 2021, so it seems to me that P06 of last summer seems to have aggrevated the problem such it finally got more obvious to someone.
The thing is that I guess in the old occurrences most of the backup files were also members of the same autobackup plan so the backups were extremely similar as only whole files were replaced. So this was not very obvious.
So for the moment we had closed the ticket and I hope I have no further findings in a few weeks/months from now.
Original Message:
Sent: Feb 18, 2025 05:03 AM
From: HZ55
Subject: IMC E710P06 corrupts configuration backup files
Thanks for letting me know. In fact I have a ticket already. My primary intention was to let the community know as crappy config backups are not a good situation.
Ivaylo just sent me the hotfix and will deploy it today. I keep you posted.
Original Message:
Sent: Feb 15, 2025 02:33 AM
From: vasilyotov
Subject: IMC E710P06 corrupts configuration backup files
Hello Holger,
I have similar case which is elevated to the developers. Yes, you are right - it happens only with auto backup plans which contain multiple switches and sometimes IMC is mixing up the backup of some of the switches during the creation and assigning of the tmp configs.
So long story short there is known issue with the auto backup plans in P06 and the developers are working on it. This week we applied the patch to my customer and currently it looks fine, but he is still monitoring the system. Please open a case with the support and ask them to elevate directly to me and also send me a PM with the case ID or just drop me a mail, so we can also apply the patch to your system.
Once we confirm it is solving the problem I think it will be merged in the new release (E0710H07)
Regards,
Vasil
Original Message:
Sent: Feb 14, 2025 11:44 AM
From: HZ55
Subject: IMC E710P06 corrupts configuration backup files
Hi folks !
All with P06 installed I do recommend to have a look at $IMCDIR\server\data\cfgbak and check the (typical) sizes of each respective device.
So you need to look at backups past the installation date of P06.
I did actually copy all to Linux and sorted them into IPaddress directories to make it much easier to sort them and grep for patterns.
The corruptions are all readable config file content. But there is a range of from
- backup file for A for contains a complete and consistent backup og B (down to the checksum)
- backup file for A contains backup for B and C or only larger parts of each.
- is still consistent - yeah happens too
- backup file for A starts with A but finishes with B
So the size is very often larger than usual. Using Linux I was using ls -ls so the initial size column
makes substantial size changes much easier to see.
It also makes sense to grep for sysname or hostname or version then names not belonging into their targetfile or multiple names in a file
will make it easy to see.
I am not sure yet whether this can also be reproduced with a single manual backup. I would suspect the mixing happens in parallel jobs
during autobackup plans but that is only a theory.
In fact we have two independent servers inhouse and both show the symptom. So this does not seeem to be a single incident matter with very isolated prerequisites for the problem to occur. Which is why I am posting here right now !
best wishes !
Holger
p.s.:
Here a sample of the sizes for one typical device which suggests something is wrong:
The first column is the size in disk blocks. This is much easier to determine as roughly the same vs the byte length of the file and still good enough for us.
total 1824
28 192.168.1.1_startup_20190709020208.cfg
28 192.168.1.1_startup_20190707020213.cfg
28 192.168.1.1_startup_20190706020208.cfg
28 192.168.1.1_running_20190709020153.cfg
28 192.168.1.1_running_20190707020158.cfg
28 192.168.1.1_running_20190706020154.cfg
28 192.168.1.1_startup_20200530023222.cfg
28 192.168.1.1_running_20200530023206.cfg
28 192.168.1.1_startup_20220216023015.cfg
28 192.168.1.1_running_20220216023011.cfg
28 192.168.1.1_startup_20220222143952.cfg
28 192.168.1.1_running_20220222143947.cfg
28 192.168.1.1_running_20220428023004.cfg
28 192.168.1.1_startup_20220428023009.cfg
28 192.168.1.1_running_20220826023013.cfg
28 192.168.1.1_startup_20220826023019.cfg
28 192.168.1.1_startup_20221112023009.cfg
28 192.168.1.1_running_20221112023005.cfg
28 192.168.1.1_startup_20230119023009.cfg
28 192.168.1.1_running_20230119023005.cfg
28 192.168.1.1_startup_20230302023009.cfg
28 192.168.1.1_running_20230302023004.cfg
28 192.168.1.1_startup_20230413023008.cfg
28 192.168.1.1_running_20230413023004.cfg
28 192.168.1.1_running_20230620023005.cfg
32 192.168.1.1_startup_20230809023010.cfg
32 192.168.1.1_running_20230809023006.cfg
32 192.168.1.1_startup_20230915023010.cfg
32 192.168.1.1_running_20230915023006.cfg
32 192.168.1.1_running_20240524023005.cfg
32 192.168.1.1_startup_20240829023009.cfg
32 192.168.1.1_running_20240829023004.cfg
48 192.168.1.1_running_20241215023006.cfg
32 192.168.1.1_running_20241216023005.cfg
52 192.168.1.1_running_20241223023005.cfg
32 192.168.1.1_running_20241224023005.cfg
48 192.168.1.1_running_20241225023005.cfg
120 192.168.1.1_running_20241226023005.cfg
32 192.168.1.1_running_20241227023005.cfg
48 192.168.1.1_running_20250104023005.cfg
32 192.168.1.1_running_20250105023006.cfg
48 192.168.1.1_running_20250106023005.cfg
32 192.168.1.1_running_20250107023006.cfg
104 192.168.1.1_running_20250123023006.cfg
32 192.168.1.1_running_20250124023012.cfg
48 192.168.1.1_running_20250125023005.cfg
32 192.168.1.1_running_20250126023009.cfg
48 192.168.1.1_running_20250128023004.cfg
32 192.168.1.1_running_20250129023006.cfg
48 192.168.1.1_running_20250201023006.cfg
32 192.168.1.1_running_20250202023005.cfg
So the file grew over time from 28 due to some changes and all the peaks are corruptions
Of course this can have false positives in rare situations and I have seen smaller ones where only B was contained. Your mileague may vary.
But good enough to have a convincing view when you have 17000 files.
Funny is now that I see it again it looks like every second may be ok. If yes, then why ? I keep you posted