Security

Hi forum,
I have a c3000 CPPM hardware appliance running 6.7.7.109. I was seeing some anomalies with event viewer where I was not getting all the logs. TAC recommended to upgrade to 6.8.4. While doing the upgrade with TAC, the eng said my grub.conf file is corrupted because of this:

[appadmin@myCPPM3000] system boot-image -l
Traceback (most recent call last):
File "/usr/local/avenda/common/bin/GrubHelper", line 178, in <module>
retVal = listMenuEntries()
File "/usr/local/avenda/common/bin/GrubHelper", line 15, in listMenuEntries
grubConfig = GrubUtils.GrubConfig()
File "/usr/local/avenda/common/bin/GrubUtils.py", line 59, in __init__
self.__parse_menu_entries()
File "/usr/local/avenda/common/bin/GrubUtils.py", line 77, in __parse_menu_ent ries
title = menu_lines[counter+5].split("=")[1]
IndexError: list index out of range

TAC edit the grub.conf file and ran the same command and the output changed to this :
[appadmin@myCPPM3000]# system boot-image -l
1) Aruba ClearPass Platform 6.7.7.109065 [Active]
2) Aruba ClearPass Platform 6.5.3.75367

Now, here is the big PROBLEM..upgrade was not working and throwing errors. TAC went back to look at the grub.conf and it went back to show the original output. TAC fixed again, and again upgrade was not happening. TAC decided to reboot the box and of course the box never came fully up. It booted up to rescue mode> .. After 6 excruciating hours of troubleshooting, mostly waiting for TAC to get someone higher. I was told by the manager that unfortunate they have to RMA the box. TAC try to blame it on hardware, but the hardware tac eng said it was not (I agreed with him too). I went to look at other CPPM we have on the environment (same model and version) and when I do the system boot-image -l is showing the same original output. I rebooted one box that is not in production and it came back without issues even thought is showing the output on the original when system boot-image -l. Basically I'm a little bit hesitant to upgrade the production boxes now given the really bad outcome. Has anyone experience this before or seen similar issue? Is the grub.con really corrupted??

Please ask TAC for the root cause analysis of the replaced unit to see if the conditions match your production systems. Grub is not something you have access to, and if the root cause is determined you have better information to determine if it is safe to do the upgrade in production. I would have the same concerns if an upgrade fails on a lab unit which is identical to the production, if you do the same, the same may happen. I'm not aware of widespread issues on this topic, as there would have been communication to prevent people breaking their ClearPass deployments.

Also, if you haven't ask for escalation of your case, support should provide you with an acceptable solution. If you send me the case number(s) in a private message, I can have a look at it.

Thank you Herman for your reply and here is more info. 

I had another two hw CPPM servers on the shelve that I need to put it online.  The company bought this two years ago - before my time.  Regardless, one came with 6.5.2 and the other one with 6.7.0. 

Before doing any upgrades, I looked at the grub.cfg file - both were showing the normal output.  I upgraded the 6.5.2 to 6.7 - first I installed the patch (no reboot required) so grub.cfg was still showing the correct output.  Then I upgrade to 6.7.0 and tada!!! after the reboot the grub.cfg file was corrupted.  I decided to stay on 6.7.12 (stable version as recommended by tac).  With the grub.cfg not showing the normal output I was able to upgraded to 6.7.12 without any issues.  I called TAC and told them my findings and this time the tac eng told me that there is a wide known issue with hardware appliances going from old versions of code to 6.7 and there is a 50/50 chance after they try to fix the grub.cfg file the box may not come back (this is what happened to me with my first box) so then I have to rma them. 

I can't RMA right now because we need all the boxes for our current project, so I'm going to stay in 6.7.12 for now. 

I hope this helps someone else in the same situation, PLEASE if you are in version 6.7.x or older before upgrading check you grub.cfg file. 

Cheers,

MLGG