Troubleshooting
The following provides troubleshooting tips to help you get back on track and includes solutions or workarounds for common errors.
Debug Logging
To troubleshoot unexpected behavior in SignServer, enable more detailed (debug) logging by running the following command in the JBoss CLI:
Start JBoss CLI:
APPSRV_HOME/bin/jboss-cli.sh -c
Run the following commands to enable DEBUG logging:
/subsystem=logging/logger=org.ejbca:write-attribute(name=level, value=DEBUG)
/subsystem=logging/logger=org.cesecore:write-attribute(name=level, value=DEBUG)
/subsystem=logging/logger=org.signserver:write-attribute(name=level, value=DEBUG)
General Issues
Transaction rolled back error
A transaction timeout can occur for requests that are taking longer time to finish than what is configured for database transactions in the application server.
Example of errors displayed in the log:
13:21:17,483 ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:8443-1) JBAS014134: EJB Invocation failed on component ProcessSessionBean for method public abstract org.signserver.common.data.Response org.signserver.ejb.interfaces.ProcessSessionLocal.process(org.signserver.server.log.AdminInfo,org.signserver.common.WorkerIdentifier,org.signserver.common.data.Request,org.signserver.common.RequestContext) throws org.signserver.common.IllegalRequestException,org.signserver.common.CryptoTokenOfflineException,org.signserver.common.SignServerException: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
...
Caused by: javax.transaction.RollbackException: JBAS014585: Transaction 'TransactionImple < ac, BasicAction: 0:ffff7f000001:672b819:5911a12e:f8 status: ActionStatus.ABORTED >' was already rolled back
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:99) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
... 100 more
Possible reasons include large file uploads and/or downloads, and/or slow network connections, slow HSM connection or HSM, or that contacting external servers takes a long time (CRL/OCSP fetching or time-stamping etc).
Solutions
- Since database transactions are not required if the worker is not configured to access the database, disabling the key usage counter, and not using any archivers will resolve this issue. For more information, see Limiting the number of signatures and Archiving.
- Alternatively, if the key usage counter or an Archiver must be used, reconfigure the application server with a higher value for the transaction timeout. For more information, see Limiting the number of signatures and Archiving.
CKR_PIN_INCORRECT (Luna HSM)
When attempting to log into a Luna HSM, the HSM may require a PIN code from an external PIN pad. If you subsequently try to log in via an application, you might encounter a misleading exception.
ERROR [org.signserver.server.cryptotokens.PKCS11CryptoToken] (
default
task-
2
) Activate failed: Failed to initialize PKCS11 provider slot
'CAAUTHDEV'
.: load failed: CKR_PIN_INCORRECT
A workaround we've seen could be to set ProtectedAuthenticationPathFlagStatus = 1 in the config file of the HSM
SQLFeatureNotSupportedException with PostgreSQL
When running SignServer with certain versions of PostgreSQL, an SQLFeatureNotSupportedException can occur on any database insert. By disabling the L2 and query cache you can forgo this issue. See Deploy-time Configuration.
P11NG / JackNJI11
UnsatisfiedLinkError
There can be a conflict between the JNA implementation in SignServer and the one installed in the system. This can be seen as errors similar to the following:
- java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
- java.lang.UnsatisfiedLinkError: Can't obtain static newInstance method for class com.sun.jna.Structure
The solution is to remove JNA from the system (i.e. apt-get remove libjna-java) or to run Java with:
-Djava.library.path=
USER_NOT_LOGGED_IN Error
If a USER_NOT_LOGGED_IN error is encountered while using the Utimaco HSM, the solution is to enable the KeepAlive setting in the Utimaco configuration file cs_pkcs11_R2.cfg
:
# Prevents expiring session after inactivity of
15
minutes
KeepAlive =
true
SunPKCS11 / PKCS11CryptoToken and Java 17
The PKCS11CryptoToken does not work on Java 17 unless the Java process that runs the application server is passed the JAVA_OPTS parameter "--add-exports=jdk.crypto.cryptoki/sun.security.pkcs11.wrapper=ALL-UNNAMED".
When using the PKCS11CryptoToken an error similar to the following may be logged:
19
:
54
:
50
,
047
ERROR [com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper] (
default
task-
1
) Method sun.security.pkcs11.wrapper.PKCS11.CK_C_INITIALIZE_ARGS.getInstance was not accessible,
this
may be due to a change in the underlying library.: java.lang.IllegalAccessException:
class
com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper cannot access
class
sun.security.pkcs11.wrapper.PKCS11 (in module jdk.crypto.cryptoki) because module jdk.crypto.cryptoki does not export sun.security.pkcs11.wrapper to unnamed module
@64dbc7bd
SunPKCS11 / PKCS11CryptoToken and certain Java 17 versions
Certain Java 17 versions/builds attempt to call C_GetInfo before the call C_Initialize, resulting in many HSM drivers responding with CKR_CRYPTOKI_NOT_INITIALIZED.The issue has been observed with OpenJDK 17 version Red_Hat-17.0.6.0.10-1.fc36 and switching to a different Java version/build is the only known solution.
12
:
30
:
33
,
581
ERROR [stderr] (
default
task-
1
) sunpkcs11: Initializing PKCS#
11
library /opt/ETcpsdk/lib/linux-x86_64/libctsw.so
12
:
30
:
33
,
582
ERROR [stderr] (
default
task-
1
) sunpkcs11: Multi-threaded initialization failed: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_CRYPTOKI_NOT_INITIALIZED
12
:
30
:
33
,
582
ERROR [com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper] (
default
task-
1
) Wrong arguments were passed to sun.security.pkcs11.wrapper.PKCS11.CK_C_INITIALIZE_ARGS.getInstance threw an exception
for
log.error(msg, e): java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
77
)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43
)
at java.base/java.lang.reflect.Method.invoke(Method.java:
568
)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper.<init>(SunP11SlotListWrapper.java:144)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapperFactory.getInstance(SunP11SlotListWrapperFactory.java:74)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapperFactory.getInstance(SunP11SlotListWrapperFactory.java:35)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getSlotListWrapper(Pkcs11SlotLabel.java:566)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getProvider(Pkcs11SlotLabel.java:120)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getP11Provider(Pkcs11SlotLabel.java:551)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getP11Provider(Pkcs11SlotLabel.java:516)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.P11Slot.<init>(P11Slot.java:57)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.P11Slot.getInstance(P11Slot.java:246)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.P11Slot.getInstance(P11Slot.java:203)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.pkcs11.P11Slot.getInstance(P11Slot.java:181)
at deployment.signserver.ear
//org.cesecore.keys.token.PKCS11CryptoToken.delayedInit(PKCS11CryptoToken.java:130)
at deployment.signserver.ear
//org.cesecore.keys.token.PKCS11CryptoToken.getP11slotWithDelayedInit(PKCS11CryptoToken.java:296)
at deployment.signserver.ear
//org.cesecore.keys.token.PKCS11CryptoToken.activate(PKCS11CryptoToken.java:153)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.BaseCryptoToken.autoActivate(BaseCryptoToken.java:115)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.BaseCryptoToken.getKeyStore(BaseCryptoToken.java:97)
at deployment.signserver.ear
//com.keyfactor.util.keys.token.BaseCryptoToken.getTokenStatus(BaseCryptoToken.java:655)
at deployment.signserver.ear
//org.signserver.server.cryptotokens.PKCS11CryptoToken.getCryptoTokenStatus(PKCS11CryptoToken.java:308)
at deployment.signserver.ear
//org.signserver.server.signers.BaseSigner.isCryptoTokenActive(BaseSigner.java:376)
at deployment.signserver.ear
//org.signserver.server.signers.BaseSigner.getStatus(BaseSigner.java:103)
at deployment.signserver.ear
//org.signserver.server.signers.CryptoWorker.getStatus(CryptoWorker.java:37)
at deployment.signserver.ear.SignServer-ejb.jar
//org.signserver.ejb.WorkerSessionBean.getStatus(WorkerSessionBean.java:287)
<...>
at org.jboss.xnio
@3
.8.
7
.Final
//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
at java.base/java.lang.Thread.run(Thread.java:
833
)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_CRYPTOKI_NOT_INITIALIZED
at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_GetInfo(Native Method)
at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11$SynchronizedPKCS11.C_GetInfo(PKCS11.java:
1681
)
at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.<init>(PKCS11.java:
156
)
at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11$SynchronizedPKCS11.<init>(PKCS11.java:
1668
)
at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.getInstance(PKCS11.java:
194
)
at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.getInstance(PKCS11.java:
166
)
...
248
more
SignClient
Cannot Produce CertificateVerify Signature
When using SignClient using a PKCS 11 token (e.g. a smart-card reader) for authenticating in some cases (e.g. both server and client running on Java 11) a cipher-suit-related error can occur:
ERROR [HTTPDocumentSigner] Failed sending request: Cannot produce CertificateVerify signature
As a workaround, SignClient can be forced to fall back to using TLS version 1.2 (as shown in the commented-out sample in script bin/signclient, a similar example is also present in the Windows batch file bin/signclient.cmd)
# In some cases, running SignClient authenticating with a P11 token
# (e.g. a smartcard reader) could give cipher-suit errors,
# In these cases, a workaround is to force the use of TLS version
1.2
JAVA_OPTS=
"$JAVA_OPTS -Djdk.tls.client.protocols=TLSv1.2 -Dhttps.protocols=TLSv1.2"