How we've identified Cyber Attacks while checking the Health of our Applications.

Performance and Security are tightly interconnected. During one of our regular performance checks, we reviewed the four golden signals, such as

1. Response times

2. Request count

3. Error rate and Exceptions

4. Utilization

Everything was within the acceptable baseline, except there were some suspicious exceptions. We've zoomed in by following the Trace from the user's click to the app and database layer.

The Trace contained the following error:.

The request was rejected because the URL contained a potentially malicious String "//"

Sample Trace we've investigated:

Below a trace from #Dynatrace

Topology
Service	Tomcat/localhost
Request	/remote/fgt_lang
Host	GBM
Process	SpringBoot
Operating system	Linux - Ubuntu 22.04.1 LTS (Jammy Jellyfish) (x86)
Bitness	64-bit
Service type	Web request service
Service main technology	Apache Tomcat
Technology	MongoDB Client (driver-core-only, QOS Logback , Java , Apache Tomcat


Metadata
URI	///remote/fgt_lang?lang=/../../../..//////////dev/
HTTP method	GET
Response status	500 - Internal Server Error
Context root	/
Server name	Tomcat/localhost
Client IP	xxxx
Thread name	https-jsse-nio-8443-exec-28
Thread ID	81
PID	1606829


Request headers
host	54.202.133.52:8443
user-agent	python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-1160.el7.x86_64
Trace ID	9f852cdba55482bd4f7232a31d194438
Span ID	4414e26be16058e1


Message:
The request was rejected because the URL contained a potentially malicious String "//"
Show affected entry in 'Code level' tab
Stacktrace:
org.springframework. Security.web.firewall.StrictHttpFirewall.rejectedBlocklistedUrls(StrictHttpFirewall.java:535)
org.springframework. Security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:505)
org.springframework. Security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:196)
org.springframework. Security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1787)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:833)

The stack trace explained that this was a cyber attack on our application. Our spring boot http firewall caught this problem.

More Details about this issue CVE-2018-13379

Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')

https://nvd.nist.gov/vuln/detail/CVE-2018-13379

Lessons learned

1. Be open-minded and investigate exceptions

2. Performance tracing helps to find security attacks

3. Security and Performance teams should work hand-in-hand

I wish you happy Performance & Security Engineering!

#Dynatrace #Security #Performance