Network Management

Was about to update AirWave and noticed a nice little warning message that the CLI would no longer be available.

At least it was in the Relase Notes, unlike the whole adding docker to AirWave.

Looking at 8.2.4 User Guide Appendix B it looks like the replacement to the CentOS shell is a choose your ending number menu.

Way to suck...

At least give me a number option to drop back out to a real shell.

Thanks for the feedback.

There was much debate that went into this feature, and this is just the initial roll out.  As we continue forward, we're hoping to develop custom modules to restore some of the functionality lost by not having direct access to the shell.  If you could help us distinguish which CLI operations you commonly perform from the CLI, we can start to plan out improvements.

We already have a request for a subset of network debugging tools: ping, traceroute, tcpdump, nslookup.  And we're keeping a watchful eye on all inbound requests.  So expanding on your feedback response would help a lot in shaping the future of the product.

A way to escape to the full shell would be much appreciated.

Menchini, again, what needs do you have critical to the operation of the AMP do you need root access for? 

Menchini, automatic daily SCP of nightly backups are there now, RFE for adding Windows shares. So if you are archiving your backups to linux now or an SCP server, you can do that today and have the script change the name once it lands on the target server. 

I'd suggest to log into innovate.arubanetworks.com and file the suggestion.

You might reference NMS-I-810 , which is a request to make this security

feature optional.

I'd like to add a request for a full shell menu option.  

Do I need to add that to the idea's feature request? 

There will not likely be any addition to the AMPCLI to enable full root access, it defies the purpose of putting this in in the first place. 

A list of what you need root access for that is critical to the operation of the AMP server would help us understand your needs and requirements, or address them via the other alternative paths provided in the GUI or AMP CLI, and if they aren't there, we can add them. But root access, just to have it, is not likely going to be a valid reason. 

One other use for the CLI.

I didn't see how you are able to load a Web https certificate for AirWave now. 

Was there a feature in 8.2.4 that allows us to change the web cert in the GUI? 

Web certs are now required to be pkcs12.  once uploaded, you can apply the cert using the Add SSL Cert option under the Security submenu.

Under the Backup submenu, there's adding a destination server.  This will automatically scp the nightly backup to the destination.  Only nightly backups, not the forced on-demand backup.

Will you be adding a CIFS option and not just a SCP option for the backups? Many are only using Windows fileshares and do not have a SCP destination. 

CIFS option would be a feature request.  If you had a script/cronjob running in the background that was performing this task, it should still be active post upgrade to 8.2.4.  Disabling of previously configured cronjobs and custom scripts will be taken care of in support cases.

We manage our own Airwave server (not an Aruba appliance) and need CLI access to monitor system health and to perform other management functions which are not available via the GUI, such as
1. Monitor individual SSD health using SMART statistics and RAID statistics and send weekly email reports of SSD status.
2. Local IP tables to allow access and disallow unwanted access.
3. Copy nightly backups to remote storage.
4. Periodic purge and rebuild of the database.
5. Weekly restart of AMP services (to prevent swap space from filling up, etc.)
6. Various other ad-hoc monitoring of the server to verify system status and troubleshoot issues like access to APs, controllers, SNMP issues, syslog issues, etc.

Several of these (e.g. SSD health and copy of nightly backups) are done via CRON, not manually. We don't specifically need root access, but some of these (iptables) require at least sudo.

1. Can look at filing an RFE to add advanced disk diagnostics to the 'Performance' page.

2. I thoguht we had an allowed networks list, but apparently not. That should be an easy RFE to file on the 'Networks' page to get parity with CPPM

3. This is already added in AMPCLI

4. This should not be necessary

5. Also should not be necessary.

6. More troubleshooting support will be added to AMPCLI over time

Hi Jerrod, I'd like to throw in another vote for scripting or cron management in some form or another, as we cannot upgrade without access to modify the custom scripts managing our ArubaOS controller via "on_controller" commands.

Support had some suggestions on how replace the scripts with some configuration changes on the controller, but I fear those adjustments will simply confuse my users.  The only real solution seemed to be configuring a device simply to remote into my controller for scripting.  I'd prefer to limit how many systems I need to manage my Aruba infrastructure (seems to be 5 separate UIs I need to access across 3 devices).

I'd prefer to have a full shell available for when I need it. And so far we've needed it. We couldn't have deployed AMP 8.2.3.1 without a root shell, because of the need to resize the underlying volumes and filesystem. 

I wish I had know that this was the direction Aruba was headed with their products, before we bought them. 

Re-scaling the disk is usually handled via creating of a new AMP deployment with the proper disk settings and then restoring a nightly backup on to the new platform. While some can properly re-partition and format, most fail and leades to big support efforts and usually loss of data. 

Jerrod, 

I don't follow. What is deploying with proper disk settings? How would I deploy the OVA a second time and get different results? 

Generally, you would use the OVA for demo, and then for a production installation you would install from the ISO and manually configure a CentOS AMP VM with the required settings and resources outlined in the sizing guide. 

To add, the AW installation guide highlights (Table 1) that the OVA is optimized for up to 100 devices. Otheriwse to use the ISO.

I wish we would post up multiple OVAs pre-configured, that may happen at some point later on. 

---

@jhoward wrote:

Re-scaling the disk is usually handled via creating of a new AMP deployment with the proper disk settings and then restoring a nightly backup on to the new platform. While some can properly re-partition and format, most fail and leades to big support efforts and usually loss of data. 

---

I have resized Airwave disks numerous times via the CLI without any issues whatsoever. Isn't it just a matter of not messing up any of the commands?

These work magic by the way (well right up until 8.2.4 that is):

lvm pvcreate /dev/sda#
lvm vgextend "VolGroup00" /dev/sda#
lvm lvresize -l +100%FREE /dev/VolGroup00/LogVol00
resize2fs /dev/VolGroup00/LogVol00

Let me see if I can help address some of these comments.  I'm also fwding to PLM so that they can review:

@jeff.g.kier - snmpwalks = US16804

@alow - change graphics - need more info, which images are you changing?  I've filed US16805, but need more info on the specifics that you're changing for PLM to make an educated decision.

@HLavender -

cert generation = US16631 (w/ CSR), US14594 (just cert generation, no CSR)

there's no SCPing of files between 2 AMPs.  If you have Failover, it will still get the backups of the monitored AMPs via http/https.

@Michael_Clarke - static route = US16801

@gilmorrr -

backups -> see the backup menu, there's a 'set destination' that will auto SCP backups to a destination.  If you already had a script performing this in a cronjob, that cronjob will continue to run.  Edits to cronjob will be mitigated in support cases.

db access -> if you can expand on the tables you utilize the most, then we can make sure the APIs get expanded to cover those tables

mass import = US16806

@Chris F

ifconfig / bonded interfaces = US16625

@Sascha Becht

db access -> if you can expand on the tables you utilize the most, then we can make sure the APIs get expanded to cover those tables

Rob

We often change the following on some of our AMP servers.

/var/www/html/images/theme/airwave/title.png

/var/www/noauth/theme/dev/login-logo.png

/var/www/noauth/theme/airwave/login-logo.png

We use have been using these appliances in a manged services enviroment for a number of years and have given cusotmers access to the AMP appliances to run reports etc. So we have changed some of the images to reflect the platforms function.

Its a small change but they need to be updated every time we patch the appliances or deploy a new one.

thanks

Andy

Got it, added the details into the feature request.  PLM are planning to review the items filed this week.

I miss being able to see which process are up - especially after a reboot.

As a power user of nearly every system I encounter, I like the ability to get to a shell - I want one in CPPM as well.

The thousand or so times I've needed to have or give shell access to Airwave for TAC, never mind system admin tasks make it seem logical to provide a shell.

You've asked us to justify why we should have it back, but I haven't seen an explanation of why you took it away.

Can you shed some light on it for me?

---

@msabin wrote:

 

You've asked us to justify why we should have it back, but I haven't seen an explanation of why you took it away.

Can you shed some light on it for me?

---

PLM should probably be the ones to respond to this.  Fwded to the PLM team.

Hi Matthew,

We removed the root access because our most security-concious customers view this capability as a security vulnerability.  And they're right about this point -- a user (or bot) with root privs can do essentially anything on a server, including malicious activities. 

We also recognize that our customers have been able to do a lot of great things with the privileges.

In prepping for 8.2.4 I talked with customers, with support and with our account teams to prioritize the most important things that users do at the CLI.  I know that we didn't implement everything.  

My plan is to continue adding to the CLI feature set to help you accomplish more of these things.  In reading through this thread and getting feedback through other channels I know that customers want to a bunch of things including: 

- Increase disk size

- View/control processes

- Transfer files

- Update files

- Test device connectivity

We are looking into doing all of these in 8.2.5.  In the meantime, anybody who wants this or other can request (via the CLI menu + TAC) that we add a custom menu item.  

Thanks,

Dan Comfort

Product Manager, AirWave

Thanks Dan

I would have liked to be one of the customers you talked to ahead of time, and I would have liked to know before hand what was happening.

I completely agree that root access was a bad thing.

I however think that no visibility under the hood makes the appliance less secure - I'm now completely having to trust Aruba to secure the OS and can't easily check when my PCI assessor asks if we're patched.

I'll learn to live with it as I need the tool more than I need to have full control.

Clearly this is not to address customer security concerns, as we suggested to PLM serveral times that if a customer requires this there is simily an option to turn it on (similar to how FIPS is implemention on ClearPass or on the Controller).  We have serveral customers who use custom routines on the linux shell of the server.  Aruba/HPE clearly wants to shut down this access.  Security is not the answer...

Hi @ all,

i am new at Airwave. i will miss the cli, too. We use Redhat here, so my first idea was to install Airwave to one of our RHEL Systems. So i could use our standard backup tool HP Data Protector. 

i think, it should be my decision which Linux, i will use for Airwave. With Airwave on our RHEL i would have a perfect integrated Airwave System. 

Greetings 

Christian

PS: Sorry, no english speaker

Changing sides of the fence - I completely agree that remote root access is a terrible security risk. I prefer the model where a general user login with sudo permissions can greatly improve security of the system.

I forgot one point: Icinga!

I do have to agree with @fredw his post here though. This could have been made optional as an added security feature. Or another solution would have been to implement two factor authentication. 

The Airwave upgrade procedure already requires your Aruba credentials to download images, something similar could have been implemented for root access. 

It kinda makes me wondor if this was done to prevent customers from screwing things up via the CLI. You can easily mess up your entire Airwave installation with 1 wrong command after all.

That being said, if Aruba manages to add the options listed here in 8.2.5, the closed shell should become a lot more practical already. 

We would appriciate it if you had the 'additional functionality' in place prior to removing the regular functionality.  Many of us use OUR ( not your ) servers for other purposes and you have just removed our ability to levergage OUR hardware the way we see fit.   I do not know any of your customers who are happy with this change.  It has cerntainly pushed us to look for other solutions.

Hi,

Any news about include vmware tools support ? because it is complicated to ask to TAC for each upgrade of AirWave (when there is a Kernel Update !)

So, today I opened yet another TAC case for 5 new issues that we are facing in light of this horrible decision.

Let me know what Arubas though on these are:

1. We need to add more routes to the Airwave server. We are not able to do that when the root access was removed. 

2. We are unable to add additional network interfaces to the machine. Was easy to do before, now it is a problem.

3. How do we install the certificate and generate a Certificate signing request? This is for the web interface.

4. When the upgrade was made to 8.2.4.1, backups stopped running. They stopped completely. We previously used a bash script putting the files on an ftp server after backup was done.

5. Still, SCP file uploading/downloading does not work. Yes, TAC has looked into this has has no solution. Tried linux server, TAC recommended ones etc. Nothing works. Works with other clients/services. Not Ariwave

mholden, while we appreciate the deep, insightful feedback of "Way to suck", it would be far more helpful to inform us on how you are using the CLI and why you require access to the root shell.

Access to the shell has causedd no end of support issues long-term where changes were made, packages installed that resulted in stability issues, modifications to settings made that created long, difficult to pin down root-causes that were impacting customer perception of the product. Additionally, leaving root access was a threat vector of bad actors having access to the system adding in software that exposed the machine to additional risk (packages that added CSS vulnerabilities and exploits via some web interface installed as part of the 3rd party package).

No other product in the Aruba portfolio leave access to the shell open and this is just the evolution of AirWave into a more secure and stable product. While we can appreicate a small handful of customers that like having access to the shell, it's not a requirement and so long as our customers are informing us as to what they need from the shell that they aren't getting from the AMP CLI and GUI, we can add those support features in as needed. 

Thanks, as always. 

Jerrod sorry, you are correct, knee jerk reaction. 

While full root access is not strickly needed, shell access has been VERY useful.

I've always hated loggin in a root, a user accuont is much better, and I've in some cases created such an account. 

Shell access is VERY much appreciated and used. 

I've used it as everything from a make shift SCP server so that we can get flash backups off the controllers, to a jump box for being able to change the default route on controllers. 

Root access has also been required when creating mount and backup scripts in order to have AirWave backups go to CIFS shares. sudoers would be fine for these functions. 

Another reason for shell Troubleshooting and Upgrading. 

Perhaps a balanced approach of going therough a couple of menu options to get to a user shell, and doing the one time key to get full root access like on the controllers would allow for flexablity while reducing the support calls.

Some of those have been discussed to be added in future versions (adding a module to enable offline backups or to make the scripting of backups easier). Downloading of the logs should contain all the troubleshooting logs moving forward (and if not, TAC case will file a bug to add that in). A jump box, while handy (and I've used it many times), it's not a feature we support or advocate so losing it shouldn't be a critical loss. 

SCP server host still works with the AMP CLI, you can load up and down files into the AMP via the AMP CLI to use for controller firmware, move files in and out, load certs and new packages, etc. 

However, there will not likely be any steps or processes that ends up with the ability to get any root access to the system, via user account, sudo, or otherwise. Thanks for the feedback, we will bring this up with PLM and engineering as things to add in. 

A few of the things I have used CLI access for:

1. Changing timezone (there was no GUI option for this in previous versions, don't know about 8.2.4).

2. SSH from Airwave server to Aruba controllers and switches has been valuable in situations where TAC needed quick access.

3. Keeping VMware Tools updated (our server guys are always bugging me about this).

Jerrod,

For me, the real issue here is the order that aruba used to make the change.  Instead of pulling the feature that people used and then asking what was needed, it would have been better to reverse the order.  Find out what tools people needed and then have them available when the feature was removed (or at least have a timeline for the features getting added). This product has been this way for a long time and there should have been a way to get this information before making this change.  

JUst my opinion.

Michael

I agree with Michael_Bloom. A change like this should be communicated to all customers well in advance, at least 6 months. You should use that period to collect feedback about what shell features are needed and used.

I also periodically use the database CLI to make queries - is there a way to access the database from an external reporting tool?

Hi mholden,

after the amp is upgraded do not log of. Before you close the cli make the following change.

change the file : /etc/passwd

change the first line to: root:x:0:0:root:/root:/bin/bash

safe the file and you can still login to the airwave with your root user

Note that if you choose to perform such actions like editing core permissions above, that you create a potential security hole, so run this at your own risk.  Also, if security compliance is a requirement, most current scans fail when root permission accounts are available, so keep that in mind if security is a priority in your network.  And it's likely something you'd have to remember to perform at each upgrade.

Hi Rob,

you're rigth please do this at your own risk.

please also keep in mind that there are no permission changes. Only the root default login directory is changed.

If there is a known root security issue please report this to the centOS development team (www.centos.org)

I know why aruba is acting like this. It is a big security risk if a admin have access and they don't know what they do.

It might also be usefull to have an option to install/upgrade vmware tools for ESX based installs.

Also not a big fan of this new menu based CLI. Question, do I now need to make a TAC case to perform the following?

- Expand the disk size > this is quite a common problem/request with our end customers

- Check running processes or disk space (top & df -h) > makes it easier to spot issues or a full disk

- Change the timezone

This is all useful feedback.  The Product team did communicate with several customers when building up the plan for the feature.  And the feature has been shown at Atmosphere, Aspire, and Discover events.  While there could have been more of an announcement of the feature, the upgrade notice does allow you to bailout if you do not want to run AMPCLI.  The custom modules structure in the menu is designed in a way that we are going to be able to address some of the requests sooner than the next release.  This was a 1.0 feature release, so there's room for improvement.

That said, let's move forward by making requests for enhancements and modules that address options that are now missing.  I've been adding them as feature stories, but it doesn't guarantee that it will make it into the scheduling - but like all features, the more requests for a feature - the more likely it'll be implemented.

For tracking:

US16673 : option to update timezone

US16776 : vmware tools installation

US16788 : expand disk size

US16789 : CIFS option for backup transfer to windows fileshare

Workarounds:

For processes, you can request through support for the 'process list' module -> while it's not top, it does show the running processes.

For disk space -> you can use the Disk Usage alert (it has an email option as well)

No longer a usage case from AirWave server:

 - Full shell access

 - Allowing CLI user to SSH into another server or networked resource via AirWave server

:: REASON ::

The above 2 items are being caught by security audits as security vulnerabilities.

For things that don't have a workaround, a support case would need to be opened to address the changes needed and help track the module &/or feature requests as they come in.