Network Management

Hello everyone,

Today i have IMC (e605) which is installed on a VM with Windows Server 2012 R2.

I have to migrate this soft on another VM and the new one is on Windows 10

Since 2 days i'm trying to install the new version of IMC (e705) on the new VM but i have a problème to start it, the process JSERVER don't want to start so i can't use IMC on the new VM and access at the web portal.

For informations, i let IMC install the Microsoft SQL 2016 that it propose.

Have you an idea about this problem please ?

Thanks for your help !

Hello,

This sounds like the known issue with jserver startup. I've posted it before here, but here is an updated text:

ISSUE: Jserver process is permanently stopped/does not start after upgrade to E0703 completes successfully.

1) Open Windows CMD as Administrator and navigate to "iMC\client\bin"

2) Run the command: cacls jmx.password /P [admin group name]:F // replace [admin group name] with the non-English equivalent of the Administrators group depending on your OS language (see also https://www.microsoft.com/en-us/language/search )

3) Restart IMC via DMA and the "jserver" process should now start up successfully

• Note: If the "cacls" command is not found, you need to make sure that "Windows\system32" is part of your command-line %PATH%, as cacls is a Windows file system permission editing command.
• The end result should be the local Administrators group having Full Control over the jmx.password file. Do NOT add any additional permissions for more users/groups to the file, or the jserver issue will remain!

At first, thanks for your help.

I'll try this but what do you mean witch "admin group name" and what product did you choose for the language on the Microsoft Web Site ?

Hello,

It depends on the language settings of your OS. For example, German-language OS uses the local Administrators group name "Administratoren" so the command would be: cacls jmx.password /P Administratoren:F

Unfortunately it was not work, the jserver doesn't start....

Can you please try these steps to debug why it won't start?

1. Edit $iMC\client\bin\startup.bat file.
2. Locate the last line, change " JAVA_HOME \bin\javaw.exe" to " JAVA_HOME \bin\java.exe", then save it.
3. Open a windows command line 
4. cd IMC\client\bin
5. Save startup output to 1.txt: startup.bat >1.txt
6. Share the output of $iMC\client\bin\1.txt after the startup fails, as well as any output from the CMD window
7. Edit $iMC\client\bin\startup.bat file  and return to " JAVA_HOME \bin\javaw.exe".

I did these instructions and this is what the command return :

C:\Program Files\iMC\client\bin>startup.bat>1.txt
OpenJDK 64-Bit Server VM warning: Ignoring option PermSize; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
Error: Password file read access must be restricted: C:\Program Files\iMC\client\bin\..\bin\jmx.password
jdk.internal.agent.AgentConfigurationError
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:590)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:436)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)

The file 1.txt which was created in the folder bin is empty.

Hello,

That's precisely the startup error that is mentioned above related to jmx.password. Could it be that your local Administrators group has been renamed?

It's likely this would also trigger the issue, as the command IMC runs on E0703 is the "cacls" command mentioned above, using "Administrators" as the group name.

If you are having trouble you could always check the local Users & Groups on the server, or run net localgroup | find "Adm" or similar to locate the Administrators group.

Hello,

One more important bit I'd like to mention... I've seen some folks writing that they have issues despite running "the command correctly" when in fact they were running "icacls" instead of "cacls". Ignore the obsoletion message of cacls, it is not exactly the same as icacls and Engineering have stated that "cacls" must be used here. The same command using "icacls" may not work as expected.

Hello,

I tried to start the service with the local administrator but the same happened.

For me in France, the admin group is named "Administrateurs" so i used the correct command "cacls", and not "icacls", like this :

cacls jmx.password /P Administrateurs:F  then the command asked to me if i'm sure of my choice and i did "yes".

But it does not works neether... I wanted to see how works the last installation in E603 and i saw that i don't have the "jmx.password" file. Is it a new features with the nw version ?

I don't know what i can do to make the soft functional :')

Hello,

Can you please open Properties for the jmx.password file, and share a screenshot of the Security tab? If you are logged in with some user that is not in local Administrators group, this should show that you do not have read access to the file. Otherwise it shows all users/groups with permission to access the file.

The reason for the jmx.password issue is related to the OracleJDK to OpenJDK 11 backend migration in IMC. This resolved a long list of security issues, and one of them was related to the JMX console being accessible remotely without a password. To secure this and prevent the vulnerability, a jmx.password file must be added which has access restriced to only the local Administrators group. That's not something our developers decided, but the folks who worked on that for (Open)JDK.

See for example https://docs.oracle.com/javadb/10.10.1.2/adminguide/radminjmxenabledisable.html

There are the autorisations for administrators and users (my profile is in the administrator group)

Hello,

Thanks for the screenshot. I see there are additional permissions in there. Please completely remove all user/group/system privileges on the file that you see on security tab, except for VDRLT622\Administrateurs, and then try starting up jserver again.

Hey,

It finally started ! It was just a problem of group of permissions...

I'd never guessed that i juste have to delete the uselesses groups will fix it

Thank you a lot for your help !

Hello,

I have the same problem, and all the options here already tried. Is there another way to solve the problem? In any case, I also get the error as above time when I rewrite the javaw.exe on java.exe.

"Error: Password file read access must be restricted: C:\Program Files\iMC\client\bin\..\bin\jmx.password
jdk.internal.agent.AgentConfigurationError"

Also i have correct the permissons on the jmx.password - but all this not work for me.

Thanks in advance

Hello,

Can you please post the same screenshot as above, showing your jmx.password "security" tab?

Hi Justin,

Same issue over here, ran the cacls command to change persission on that file, upon rebooting the server I noticed that the permissions still had another entry, this cannot be removed at all, you remove it and it comes back automatically, any further troubleshooting?

cant upload images for some reaon but the group is called ALL APPLICATION PACKAGES

Running your diag I get the following which suggests the file is still at fault "Error: Password file read access must be restricted"

Matt

Hello,

The jserver workaround was proposed by Engineering using cacls, hence my recommendation. However I've noticed that it cannot overwrite the permissions that are inherited. After some experiences removing the permissions, it's actually not absolutely necessary to use cacls. Here are alternative steps that should be easier to implement and account for this...

I simulate the issue on English Windows with hostname IMC by opening "lusrmgr" (Local Users and Groups), and renaming the Administrators group to "RenamedAdmins". This will trigger the issue if I install IMC afterwards.

To fix it:

1) Open Properties > Security on the IMC\client\bin\jmx.password file

2) Notice that there are probably 4 different permissions shown - click the icon to get Administrator permissions if you cannot see or edit the permissions. ALL APPLICATION PACKAGES, SYSTEM, Users and RenamedAdmins (this will be different based on the name of your local Administrators group). These are the inherited permissions that cause issues.

3) Click on Advanced, then click on Disable inheritance and select Convert inherited permissions into explicit permissions on this object (selecting the other option will remove all permissions, including your own, so don't use that). Your Advanced Security Settings should now look similar to this:

4) Remove all permissions, except the second one on the list above (the permissions for the local Administrators group, called IMC\RenamedAdmins in my case), like so:

5) Click on Apply to save it, and close the Properties. Start IMC again, and jserver process should now start up successfully.

Hope that helps.

Hi,

Also in french language, the solution to only keep Administrators (Administrateurs in french) group ACL worked for me too. I only said one thing, without java, life would not be so funny...

Hello Justin

I removed inheritance and clicked the convert to explicit option when presented, I still cannot remove the group "ALL APPLICATION PACKAGES" 

And I still cant start JServer process from DMA

Matt

Hello Matt and all,

Is it possible that you are not the owner of the file and therefore can't remove it? I can't see the owner in the Security context screenshot.

Please have a look at this doc (rather old but should still be relevant) from Oracle on securing jmx.password in Windows:

https://docs.oracle.com/javase/8/docs/technotes/guides/management/security-windows.html

Another possibility is that Group Policy is somehow interfering with the permissions, if you are in an AD Domain. See for example the steps mentioned here:

https://support.microsoft.com/en-us/help/2798317/microsoft-store-apps-fail-to-start-if-default-registry-or-file-permiss

In that case, you could test assigning the system to a temporary OU that blocks GPO inheritance and try again after updating the policy on the server.

Would you know where any log files are put when jserver fails to start? I'm also having this problem trying to set up a trial of IMC. 

I'm running on an english OS without the administrator's group renamed, so I never did try the command, but looking at the permissions, I also seem to be having the "ALL APPLICATION PACKAGES" permission come back on. This may still be my problem but I would also like to know if my log files would show me anything different, and I don't know where to find them. 

Thank. It helped.

Hello,

There is no IMC log to debug the jserver startup, because jserver doesn't "log" anything until it actually starts up - which it won't if the permissions do not work out. The way to "debug" the startup is mentioned in the 5th post or so further up here - you can rename the client\bin startup.bat javaw.exe to java.exe, and then attempt to run startup.bat from CMD.

This will print some output, which will almost certainly be an error about jmx.password security.

EDIT: Removed the workaround of last resort, please do not use it. New Platform versrion iMC 7.3 E0705 has been released and resolves the issue in this topic entirely.

@jguse 

i followed this whole topic and read also all messages including the recommended workaround concerning the inheritance and all the stuff.

i also tried your highly insecure workaround with the renaming of the both startup.bat files on a test system (because all other recommended workarounds didn't work.)  and its worked as you said.

So basically i am interested in the reason why your workaround, described as highly insecure, is highly insecure and on top of that if there exists a solution right now.

So i am not the only person you might have noticed; so it would be nice if there is a solution

Hello,

I think all possible/known solutions to this issue have been posted throughout this topic. I've also posted earlier how to debug the jserver startup (modifying the javaw.exe to java.exe in startup.bat) and checking the output on CMD - have you tested this when using the "normal" startup.bat? If so, what output do you see in the CMD window?

The reason why startup_d.bat (which exists for debugging purposes) works, is because it starts the "java server" aka jserver process of IMC without any of the security and other options. This can be seen by comparing the two startup files. Startup_d remarks (comments out) these lines:

rem Enabling Local/Remote Monitoring and Management
rem set JMXRMI_PORT=9091
rem set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote.port=%JMXRMI_PORT% -Dcom.sun.management.jmxremote.authenticate=false -Djavax.net.ssl.keyStore="%IMC_HOME%\security\newks" -Djavax.net.ssl.keyStorePassword=iMCV500R001

This means the SSL keystore is not set, there will be no authentication on the jmx.remote agent (which means a JMX console on IMC can be accessed remotely without login), and the jmx.remote port is not set to 9091. The port not being set might cause issues with distributed installations, though I didn't test it specifically and can't confirm.

The "Java Heap Size" is also set to 512MB by default, which will make IMC run very slowly (if at all, on larger installs). You can fix that though, by modifying the -server -Xmx512m in the following line:

set JAVA_OPTS=-server -Xmx512m -XX:PermSize=64m -XX:MaxPermSize=512m -Dimc.home="%IMC_HOME%" -Duser.language=en -Duser.country=US -Djava.io.tmpdir="%IMCROOT%\tmp" -Dorg.apache.el.parser.COERCE_TO_ZERO=false

Here is the same line on my test system with 8GB Heap Size (out of 16GB total memory), and a few other changes that only exist in the "normal" startup.bat:

set JAVA_OPTS=-server -Xmx8192m -Xrs -XX:PermSize=64m -XX:MaxPermSize=1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../log -Dimc.home="%IMC_HOME%" -Duser.language=en -Duser.country=US -Djava.io.tmpdir="%IMCROOT%\tmp" -Dorg.apache.el.parser.COERCE_TO_ZERO=false -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true

If you are still struggling to get the proper startup.bat working, please share the output I mentioned above from CMD, and open a support case if it's urgent.

As follows, you see the output of the debugging:

output from cmd

C:\Program Files\iMC\client\bin>startup.bat >2.txt
OpenJDK 64-Bit Server VM warning: Ignoring option PermSize; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
Error: Password file read access must be restricted: C:\Program Files\iMC\client\bin\..\bin\jmx.password
jdk.internal.agent.AgentConfigurationError
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:590)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:436)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)

the outfrom from the 2.txt file is empty....

Hello,

This error provided matches the issue I've described in previous posts here.

It means that the security settings on your jmx.password file are not correct. This file must be configured with Full Control for the local Administrator Group and absolutely no other permissions.

If you are unable to figure it out, can you please post a screenshot of the jmx.password file Security settings (permissions)?

i am absolutely capable of configuring the jmx.password file concerning the mentioned security settings, cause it is my daily business.

But as i said before, it is not possible to delete the specific application group (the following picture is in german)

So it is like all the other people before described this issue.

The jmxpassword file is configured with Full Control for the local Administrator Group but it is definitely not possible to delete the other rights (Allen Anwendungspakete)

So what should i do next?

Hello,

I've been unable to reproduce that particular issue in the Lab, so I am not sure how exactly to solve it, and support cases I've seen got resolved by removing all the excess permissions. That's why I've suggested to double-check the permissions, and sorry I didn't realize you had already done it and got stuck on ALL APPLICATION PACKAGES.

I am not sure why you can't remove this permissions, and have not found too many good explanations for this behavior online. However there is this post.

We have been informed by Microsoft that this behavior is "by design".

ALL APPLICATION PACKAGES supports UWP apps and if you assign it permissions, Windows then attempts to "protect" them.

So, not a bug. An undocumented feature!

It's for Windows 10, and I don't know if that 'feature' applies to Windows Server as well.

An article from Microsoft here about a different issue (where this permission is actually missing) and they recommend steps there to check if permissions were manually configured in the Registry or via Group Policy.

Maybe the steps mentioned there can help you figure out if you have Registry or Group Policy settings that might be applying the permission. Group Policy is probably the most likely candidate... quoting the article:

Determining if Group Policy is being used to manage permissions

1. Logon to a PC as a user experiencing the problem
2. Open an administrative command prompt then run the following command:
   
   > gpresult /h <path>\gpreport.html
    
3. Open the file gpreport.html and expand the following path:
   
   Computer Settings -> Policies\Windows Settings\Security Settings. look for "File System" and Registry. If these exist then GP is assigning permission. You must edit the GP to include the necessary permissions for the All Application Packages group.

Can you check if such settings might be applying permissions?

Hope that helps.

Hello, I am sorry that I will contact you only now?

Here is the screenshot ....

The Group "Alle Anwendungspakete" can not be removed.

This is always newly created

Best regards, Stefan

Hello HirnerStefan

When you are on this window, go in the "Advanced" setting at the bottom of this page.

Then disable the "inheritance" at the bottom again.

When you try to do that, a message with 2 choices will appear, take the first one.

Now you can validate and i think you could delete the autorisation and just keep the administrator group

This worked for me! Thank you

Best,

Renato

Hello everyone,

Good news, iMC 7.3 E0705 has been released in the meantime, which is not affected by this issue and should resolve it for you when you upgrade. Please open a support case if there are still issues with the jserver starting post-upgrade.

iMC Standard: https://h10145.www1.hpe.com/downloads/SoftwareReleases.aspx?ProductNumber=JG747AAE

iMC Enterprise: https://h10145.www1.hpe.com/downloads/SoftwareReleases.aspx?ProductNumber=JG748AAE

If you were previously using the last resort workaround with startup_d.bat, please revert to the original startup.bat.

I have this same issue and tried resetting the permissions for jmx.password as per previous suggestions but had no success.

After downloading the latest E0705 update, it would fail part way through the install and the dma log would show an error with the jmx.password file again so I just set full permission for Everyone and it finally finished the upgrade successfully.

On a domain network, should the permissions for this file be domain\Administrators Full Control as well as Local Administrator Full??

I can finally log into it again, hooray!

Hello,

The permissions of the file do not need to be touched if you upgrade to iMC 7.3 E0705. iMC will set the permissions correctly now during the upgrade - which should be Full Control for only the local Administrators group (and no other permissions whatsoever).

hi Justin

i see this is an old issue, i looked at this and saw that i do not have a jmx.password file.  so when trying to run that command it cant find the ile cause it doesnt exist.

Hello there,

If you don't have a jmx.password file, then it is unlikely that you are on 7.3 E0703 version that is affected by the same issue mentioned here. Please start a new thread if you have issues with jserver startup on any other version.

If you happen to be on 7.3 E0703 and affect by this, then upgrading to E0705 is the most effective way to resolve it.

If you are running a version prior to 7.3 E0703, upgrade directly (from any 7.3 E0605 version, including patches/hotfixes) to 7.3 E0705, skipping E0703 entirely.

Note that if you run Server 2008 / MSSQL 2008 / RHEL 6.x, you should perform a migration before upgrading to E0705 - please follow the IMC Windows/Linux Migration Guides included in the "manual" folder of the latest iMC software download package.