JDK Flight Recorder: How to discover code hotspots

Locally running JVM processes are listed in the JVM Browser view. You need to identify the JVM you want to profile. Besides, you can either open the JMX console for that JVM or the control flight recorder.

With OpenJDK, Flight Recorder is available remotely via JMX. You need a JMX socket configured on the JVM that you would like to profile.

There are two ways to make the JMX socket available:

The following is an example of a command to start JMX with minimal configuration. You need to know the PID of a process and be logged under the same user as the target JVM.

jcmd PID ManagementAgent.start +
jmxremote.authenticate=false +
jmxremote.ssl=false +
jmxremote.port=5555

Once the command is executed, you can connect to your JVM via port 5555 using the instruction in the following pages.

If the JVM you want to profile is behind NAT/firewall (e.g., it is running within Kubernetes), you may need to use port forwarding and additional configuration tweaks to make JMX work.

Having configured the JMX socket, you need to add a remote JVM to it in the JVM Browser view. Click the

button on the toolbar of that view. You will be prompted to enter JMX connection details. Once you finish JMX configuration, nodes for your remote JVM will appear in the view. Now, you can see either a JMX Console or control flight recorder.

What if you have console access to JVM, but no way to connect to it via JMX?

You can still record JFR files using jcmd, then copy them to your machine and open them in Mission Control.

The JMX Console provided by Mission Control won’t be useful in this case, but you can use console tools such as pidstat or sjk to monitor per-thread CPU usage.

JFR recording can be produced without Mission Control, too. You can use jcmd to create JFR files, then copy them to the desktop and open them in Mission Control.

Here is an instruction to capture JFR recording. You need to know the PID of the target JVM and execute jcmd under the same user account.

Now that you have a JFR file, you can open it in Mission Control later.

Stack view allows visualizing a set of stack traces as a tree.

How do stack traces become a tree?

We know how Java stack traces look like (be it exception stack trace or stack trace from thread dump). Data produced by sampling profiling is just a number of stack traces.

Here are steps on how this data is transformed into a tree.

In the context menu, you have the following Stack Trace view customization options.

If we look at Java thread at any given time, it could be

JFR samples only threads in categories 1 or 2. There is also separate sampling for 3, 4 and 5.

So, if your server is "starving" on CPU, JFR will show you a particular picture, but it will be skewed due to mixing 1 and 2 categories. The sampling picture will not accurately reflect real code computation cost in this situation.

If the server is not CPU "starved" and threads you are profiling are actively consuming CPU, method sampling is the right tool for you to start with.

If the server is not CPU "starved" and threads you are profiling are low on CPU, you are likely to have a "cold" hotspot, falling into categories 5 and 6.

Category 5 code can be analyzed with the "Native method sampling" event.

Category 6 is either contention or IO, examined with separate reports.

So by looking at the CPU, you get an idea which report will be most useful.

What if you have a JFR recording and have no idea of CPU usage by threads at runtime? That data is recorded in a JFR file as well.

Open the Event Browser and find the Thread CPU Load event under the Operating System > Processor category as in the following image.

CPU usage for each thread is recorded with 10-second intervals. Remember that the percentage is taken from the total number of cores. In the screenshot, 25% means 100% of a single CPU core.

JVM and host OS CPU usages are also recorded. Click the CPU Load event type in the same category.

You can now find CPU hungry threads, but the Method Profiling report shows all threads. You can filter by a subset of threads if necessary.

Mission Control has very flexible but non-obvious filtering features.

To filter by a subset of threads in the Method Profiling report, perform the following steps.

Now, the report includes events only for threads you have chosen.

Any filter is shared across all reports, so do not forget to reset it when switching reports.

While completing the steps above, you may notice that the Stack Trace view is available in Threads. Moreover, it was reacting to selection in the thread list.

How is the Stack Trace on the Threads report different from the Method Profiling report?

Each report has a scope: a set of events (usually certain types of events) used to calculate the report. When you work with report UI, a focused subset of events in scope is maintained behind the scene.

A filter can narrow the scope, and this is what we did in the previous section.

Any event may include the stack trace, and many do. The Stack Trace view visualizes all stack traces of the focused set of events in an active report.

In Method Profiling, only method profiling samples are in scope.

The Threads report includes almost every event bound to some thread (both sampled and non-sampled). So, while technically they can be aggregated together, it makes little sense.

Nevertheless, you can use filters to fix this to make the Threads report show the same picture as Method Profiling.

Now, your Stack Trace tab includes only method profiling data, and you can change selected threads without switching reports.

The way JFR takes stack traces during profiling is precise. The majority of Java profilers use a JVM wide thread dump to get traces of all threads at once.

A thread dump is a Stop-the-World operation, and doing it 100 times per second can be expensive. JFR manages to avoid Stop-the-World. Instead of stopping all threads at once, JFR stops them individually (using OS facilities) and captures stack traces from individual threads.

Entering/leaving Stop-the-World state involves sophisticated protocol between application and system threads in JVM, and JFR skips this overhead altogether.

Yet, this optimization introduces certain nuances of data captured by JFR. Only stack traces ending up in Java code (not the native code) are recorded and visible in the Method Profiling report. Thus, unless threads you are looking at the active CPU consumers, the picture in Method Profiling will be skewed.

Besides the "Method Profiling Sample," there is the "Method Profiling Sample Native" event type that captures threads in states 4, 5 and 6 (see the list mentioned earlier). However, this event type is available only in the Event Browser not in the Method Profiling or the Threads reports. The good news is that the Stack Trace view also works in the Event Browser report.

Another important caveat for all types of Java sampling profilers is the "safepoint bias." In short, due to various effects of JVM runtime and JIT compiler, profilers can incorrectly identify the exact hot method or code line. In some edge cases, profilers can be very inaccurate.

Technically, JFR avoids the "safepoint" part of the problem, though it is still biased due to the way Java JIT compiler works.

While the edge case of the "safepoint bias" effect could make you lose your trust in the profiler, it is rarely a problem in actuality. A profiler’s job is to narrow the scope of search; it does not need to be 100% accurate to be useful. But do not trust it blindly either.

This report is composed of "File Read" and "File Write" events, capturing blocking file IO operations. Since events include file names, this report shows at the file level how many bytes were read/written to each file and how much time it has taken.

An important caveat here is to consider the JFR recording session’s threshold. Only operations exceeding it will be visible here.

The report is relatively simple but can be quite powerful if combined with the filtering capabilities of Mission Control.

You can filter by focus and aspect. The Stack Trace view is also available here.

The "Socket I/O" report is quite similar to File I/O, but aggregation is shown by remote address and port.

The taskbar lists three aggregation options available.

Lock Instances reports are based on the "Java Monitor Blocked" event type. These events are produced when a Java thread is blocking, trying to acquire the semaphore.

This report is only useful with synchronized keyword-based thread coordination.

What about java.util.concurrent-based contention?

JFR also has events related to "new style" synchronization "Java Thread Park," but there are no dedicated reports in Mission Control yet.

We’ve already had a brief interaction with the Threads report. It consists of a thread list and a timeline area.

The thread list is simply a list of threads, but the timeline is more interesting.

It has lanes for each thread and could visualize a wide range of JFR events. By default, they are so-called "Java Latency" events, but you can customize that via the context menu (see the Edit Thread Activity Lanes option).

Typically, individual events on the timeline will be tiny. You can hover to get the details under the mouse pointer, but it is not very useful if events are subpixel sized.

To zoom in, drag the rectangle over the timeline and hit Zoom to Selected Range on the context menu.

You can use the time range selected in the Threads report as a filter on other reports, too.