JSON Logging & Datadog

Why You Should Use JSON for Logging

If you’re not using JSON logging today, you’re probably relying on plaintext logs and regex to make sense of them. That might work at first, but it will start to break, lose structure, and become worse and worse as systems grow. JSON logging is the standard. In modern applications, having structured, easily parseable logs is no longer optional. It provides a level of consistency across services while still having flexibility.

Datadog recommends JSON logging for a reason. When logs are emitted as valid JSON, Datadog's preprocessor automatically parses your logs and maps them to a number of reserved attributes that reduce overhead and ensures consistent field extraction. No extra configuration is needed as these reserved attributes provide the most relevant fields make it easier to integrate with other systems, filter logs, search efficiently, and set up alerts. Crucial reserved attributes include host, service, status, trace id, span id, and others, which are essential for observability and troubleshooting.

How to Set It Up

In my experience, there were an abundance of resources in how to best configure JSON logging. The primary reference I used was Datadog's documentation: https://docs.datadoghq.com/logs/log_collection/java/?tab=logback. I found it best because Datadog was the primary observability tool we used and they would be able to troubleshoot easier if any issues occurred.

Using the Datadog docs as a guide, I started by creating the logback-spring.xml configuration file. This is where most of the work happens. Initially, I experimented with various JSON encoders and added fields I thought might be helpful. Eventually, I realized the power of simplicity: a clean, minimal configuration is easier to maintain and less likely to break.

<springProfile name="json-logs">

        <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
            <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
        </appender>

        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>

After adding the required Logback dependencies and adding this code to my logback-spring.xml file, the final major step is to set 'json-logs' as the active Spring profile in your application YAML. This ensures that JSON logging is enabled across your environment and works consistently across all services.

If these steps were followed correctly, you should be able to tail your logs and see the results. They should be in a key-value log similar to this:

{"@timestamp":"2025-1-13T13:02:27.512333-02:00","@version":"1","message":"Tomcat initialized with port 8080(http)",
"logger_name":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer","thread_name":"main","level":"INFO",
"level_value":20000,"appName":"json-logs"}  

At this point, your logs in Datadog should be structured, trace correlation should work for logs with trace IDs, and you should see more detailed and consistent attributes.

Common Mistakes & Advice

• Use a file appender if that works best for your system. For our environment, the console appender suited our needs better
• Do not get caught up in the variety of JSON encoder options; simpler is usually better, at least at first
• Check for existing log processing rules in your Datadog agent. These rules can interfere with JSON logs and prevent them from being parsed or delivered correctly. Make sure legacy loggers are not conflicting with your configuration
• For more control, consider creating multiple Spring profiles. This allows you to toggle between logging levels or formats across environments without changing the application code
• If tracing isn't working for many of your logs on Datadog, this is because they automatically sample tracing for many of your logs. The JSON is working, don't worry, but you can set it to a different rate

With the right setup, JSON logging in Datadog becomes reliable and much easier to maintain. By removing legacy assumptions and keeping your configuration simple, you gain structured logs, consistent fields, trace correlation, and easier filtering and alerting. JSON logging is not just a best practice; it is essential for improving observability in modern applications.

About the Author

Like what you heard? Have questions about what’s right for you? We’d love to talk! Contact Us

Related Articles