Log4j2 Configuration: A Detailed Guide to Getting Started

I wrote a detailed guide about using and configuring Log4j2 for Scalyr a while back. Here it is.

 

We covered basic logging for Java applications a while back. In that tutorial, we used log4j version 2, a logging framework from the Apache project. Let’s go one step further with Java application logging and look at log4j2 configuration.

Log4j’s capabilities have made it one of Java’s most popular logging frameworks. It can be configured for multiple logging destinations and a variety of log file formats. Log messages can be filtered and directed at the individual class level, giving developers and operations personnel granular control over application messages.

Let’s examine these mechanisms by configuring log4j with a command line Java application.

Sample Application

Let’s start where we left off in the previous tutorial, with an application that logs with log4j.

This is similar to the application at the end of the previous post, with a few additional logging statements. We’re logging the same message at each of log4j’s predefined logging levels: trace, debug, info, warn, error, and fatal.

We will be using log4j’s YAML file format, so you’ll need to add a few additional dependencies to your pom.xml (or build.gradle).

Set this code up so you can build and run it using your favorite Java tools.

Essential Log4j2 Configuration

Default Configuration

Let’s run our application without a log4j configuration file. If you already have one, delete it or move it to another file name so that log4j will ignore it.

When we run the application, we see this on the console:

Two of the six log messages, the ones specified as “error” and “fatal,” are sent to the console.

Log4j has a default configuration. It will log to the console, showing messages classified as “error” or higher.

Knowing how log4j will behave without a configuration file is useful, but let’s look at how to set it up for our needs.

Configuration File Location

We can provide log4j with a configuration file in a specific location via the log4j.configurationFile system property. This is the first place it will look for a configuration file.

If log4j cannot find the system property, it looks for a file in the classpath. Since log4j version 2 supports four different file formats and two different file naming conventions, the rules for locating a file are complicated. We’ll go over them after we cover the different options.

Configuration File Formats

Log4j will load Java properties and YAML, JSON, and XML configuration files. It identifies the file format by examining the file extension.

  1. Java properties — .properties
  2. YAML — .yaml or .yml
  3. JSON — .json or .jsn
  4. XML — .xml

A file specified by the log4j.configurationFile system property must have one of these file extensions but can have any base name. Log4j will parse it based on the format indicated by the extension.

When log4j scans the classpath for a file, it scans for each format in the order listed above and stops when it finds a match. For example, if it finds a YAML configuration, it will stop searching and load it. If there is no YAML file but it finds JSON, it will stop searching and use it instead.

Configuration File Names

When log4j scans the classpath, it looks for one of two filenames: log4j2-test.[extension] or log4j2.[extension]. It loads test files first, giving developers a convenient mechanism for forcing an application to log at debug or trace level without altering the standard configuration.

Scanning for Configuration

When we put the rules for file formats and names together, we can see log4j’s algorithm for configuring itself.

If any of the following steps succeeds, log4j will stop and load the resulting configuration file.

  1. Check for the log4j.configurationFile system property and load the specified file if found.
  2. Search for log4j2-test.properties in the classpath.
  3. Scan classpath for log4j2-test.yaml or log4j2-test.yml
  4. Check for log4j2-test.json or log4j2-test.jsn
  5. Search for log4j2-test.xml
  6. Look for log4j2.properties
  7. Search for log4j2.yaml or log4j2.yml
  8. Scan classpath for log4j2.json or log4j2.jsn
  9. Check for log4j2.xml
  10. Use the default configuration.

Practice Proper Configuration File Hygiene

There are 12 potential configuration file names for log4j. Loading the wrong one can lead to lost logging information or diminished performance if an application logs unnecessary messages in a production environment.

Before deploying code, make sure your application has one and only one configuration file and that you know where it is. If you insist on loading configuration from the classpath, scan for spurious files before releasing your code.

Basic Configuration

Now that we know how to supply a configuration to log4j, let’s create one and use it to customize our application.

Log4j’s Default Configuration Revisited

Let’s start with the default configuration and modify our application’s behavior from there. We’ll take the hint from log4j’s configuration rules and use YAML.

The default configuration looks like this:

Create a file name log4j2.yaml with these contents and set log4j.configurationFile to point to its location.

Next, run the application. You’ll see the same output as before.

We’ve taken control of our application’s logging configuration. Now let’s improve it.

Log File Location

The first step is to get our logs off of the console and into a file. To do this, we need to understand appenders.

Appenders put log messages where they belong. The default configuration supplies a console appender. As the name suggests, it appends messages to the console.

We want a file appender. Let’s replace our console appender.

File appenders have a name, just like console appenders. But instead of a target, they have a fileName.

Similar to console appenders, they also have a PatternLayout, which we will cover below.

The name is not just for show. If we want to replace the console appender with the file appender, we need to let our logger know where to put our log messages.

So change the ref value in the logger to the file appender’s name.

Now, rerun the application. Instead of logging to the console, it places the messages in a file named logfile.log in the working directory. We’ve moved our logs to a file!

Logging Levels

We still only saw two of our six log messages in our log file. Let’s talk about loggers and how they manage log messages.

Our basic configuration defines a single logger.

It has a level of “error,” so it only prints messages that are errors or fatal.

When a logger receives a log message, it passes it on or filters it based on its configured level. This table shows the relationship between logger configuration and log message level.

Event Level Message Will Be Logged = X
TRACE DEBUG INFO WARN ERROR FATAL OFF
ALL X X X X X X
TRACE X
DEBUG X X
INFO X X X
WARN X X X X
ERROR X X X X X
FATAL X X X X X X
OFF

So if we change the level of our logger, we’ll see more messages. Set it to “debug.”

Next, rerun the program. The application logs all of the messages that are debug level or higher.

Logger Hierarchy

Log4j arranges loggers in a hierarchy. This makes specifying a different configuration for individual classes possible.

Let’s change our application and see this in action.

We’ve added an inner class that creates a logger and logs a message with it.

After Main does its logging, it calls LoggerChild.

If we run this with our current config, we see the new message and that it’s logged from a different class.

Loggers have a class hierarchy similar to Java’s. All loggers are descendants of the Root logger we’ve been working with so far. Loggers that lack any specific configuration inherit the Root configuration.

So when Main and LoggerChild create loggers using their class name, these loggers inherit Root’s configuration, which is to send debug level and higher messages to the File_Appender.

We can override this specifying configuration for the two loggers.

Loggers are named in the logger section. Since we’re listing two, we use the YAML array syntax.

We set com.company.Main’s logger to “error” and com.company.Main.LoggerChild’s to “debug.”

The additivity setting controls whether or not log4j will send messages from a logger’s ancestor to a descendant. If set to true, both loggers will process the same message. Some systems want to add the same message to two different logs. We don’t want this behavior, so we’ve overridden the default and specified false.

Now run the program again:

We only saw the error message from Main but still saw the debug message from LoggerChild!

More Than One Appender

Just like we can have more than one logger, we can have more than one appender.

Let’s make a few changes to our configuration.

Add a second file appender. To do this, create a list with the original appender and the second one with a different name and file. Your Appenders section should look like this:

Next, point the LoggerChild logger at the new appender. Your Loggers section will look like this.

Now run the application and you’ll see two different log files, each with the messages from their associated classes.

Log Message Formatting

Each of our appenders has a PatternLayout.

PatternLayout is an instance of a Log4j layout class. Log4j has built-in layouts for logging messages in CSV, JSON, Syslog, and a variety of different formats.

PatternLayout has a set of operators for formatting messages that operates similarly to C’s sprintf function. By specifying a pattern, we control the format of log messages when they are written by the appender.

Our layout string looks like this:

Each % corresponds to a field in a log message.

Format Specifier Description
%d{HH:mm:ss.SSS} date as hour :minute “seconds . milliseconds
%t thread name
%–5level log level, right-padded to 5 spaces
%logger{36} logger name, up to 36 package levels deep
%msg log message
%n carriage return

There are many additional operators for PatternLayout.

Variable Replacement

Configuration files can become repetitive as appenders and loggers multiply. Log4j supports variable substitution to help reduce repetition and make them easier to maintain. Let’s refine our configuration with the use of Properties.

At the top of the file, we declared two Properties, one named LogDir and another DefaultPattern.

After declaring a property, it can be used in the configuration using braces and a dollar sign: ${LogDir} or ${DefaultPattern}

LogDir is a subdirectory name we added to the names of the two log files. When we run the application, log4j will create this directory and place the log files there.

We specified DefaultPattern as the pattern layout for our two log files, moving the definition to one place. If we want to modify our log file format, we only have to worry about changing it once now.

Log4j can also import properties from the environment. You can find the details here.

For example, if we want to import the log file directory from a Java system property we specify it as ${sys:LogDir} in the log4j configuration and set a LogDir system property to the desired directory.

Automatic Reconfiguration

Log4j can reload its configuration at a periodic interval, giving us the ability to change an application’s logging configuration without restarting it.

Add the monitorInterval setting to the Configuration section of the file and log4j will scan the file at the specified interval.

The interval is specified in seconds.

Conclusion

Log4j is a powerful logging framework that allows us to configure our applications to log in a variety of different ways, with granular control over how different components use log files. This tutorial covered the basic aspects of configuring log4j, but there’s much more to learn. You can learn about log4j configuration on the project’s website.

To learn more about Java logging and logging strategies in general, you’re already in the right place! Scalyr’s blog has many more tutorials and reference guides like this one.

Scalyr offers a log aggregation tool, which means that once you have lots of log files and data, they’ll help you organize, search, and make sense of all these data. So stay tuned for more!

 

unsplash-logoPhoto credit: Jacob Miller

REST API: Your Guide to Getting Started Quickly

Here a post I wrote for Stackify a while ago. You can find the original here.

 

Even though REpresentational State Transfer, also known as REST, is often referred to as a protocol, it’s an architectural style. It defines how applications communicate over the Hypertext Transfer Protocol (HTTP). Applications that use REST are loosely-coupled and transfer information quickly and efficiently. While REST doesn’t define data formats, it’s usually associated with exchanging JSON or XML documents between a client and a server.

We’re going to use a simple service and a web browser to learn about the fundamentals of REST.

Setup

For this tutorial, you’ll need a system with Docker installed. You can find instructions for your computer here.

First, follow the instructions and install Docker.

Then, once you’ve completed the installation, you can download and run our sample REST server.

Finally, start the server with this command:

Docker downloads the server image and runs it. The command tells it to make the web server available on port 8080.

So, point your browser here:

http://127.0.0.1:8080/swagger-ui.html

If everything is working, you’ll see a web page like this:

REST API Tutorial Main Page

This is a Swagger page that documents the REST API published by this server. We’ll use it to demonstrate how REST APIs are consumed by applications.

Introduction to Swagger and REST

Click on the word tutorial-controller toward the bottom of the page. It will expand, and you’ll see this:

REST API Overview

Let’s try a simple request before we start examining each aspect of the API.

Next, click on the blue GET box under tutorial-controller. 

REST API Tutorial Get All

This is a description of the Get All Employees API method. The box documents the status codes it returns, the content type, and the API path. We’ll cover this in detail as we go. Let’s make a request.

Click the Try it out! button at the bottom right of the blue shaded area.

REST API Tutorial Get All Result

Swagger made an API request for us and provides a detailed description of the result. We see three employee records.

Let’s get to work!

CRUD!

The name REpresentational State Transfer implies exchanging data. The server acts as a data store, and the client retrieves and stores data. The server transfers object states to the client. The client can update these states too.

Most REST APIs implement CRUD: Create, Retrieve, Update, and Delete.

Go back to the Swagger page and click on the blue GET box so it collapses. Here’s a quick tip: at the top of the page, there is the List Operations option. Clicking there will collapse the operations into a list again.

Let’s look at the list of operations again.

On the left-hand side of the page we see GET, POST, DELETE, GET, PATCH, and PUT. These are HTTP methods that correspond to operations.

We can map these operations into CRUD.

  • POST—Create
  • GET—Retrieve
  • PUT / PATCH—Update
  • DELETE—Delete

We’ll cover each operation as we take a look at how REST APIs work.

REST API Tutorial

Create

Let’s add a new employee to the list.

First, click on the green POST box.

REST API Tutorial Create Employee

In the top right corner, we see the API method name, Create Employee. This is documentation Swagger extracts from the application code. Many REST Servers use Swagger or a similar web application to document their APIs.

Next, we see information about how the method responds to requests. A successful request will yield an HTTP Created response code. This is a standard convention for REST APIs.

Under the response information are the request parameters. This API expects a JSON employee record. There’s an example in the box on the right.

Click on the example, and the page will populate the value box on the left for you.

Let’s edit the example.

Now, click the Try it out! button.

REST API Tutorial Create results

First, at the top of the response, Swagger tells us how we could make the same request using a command line tool called curl. If you are comfortable with the command line, you can try it out. I am going to omit that part of the responses going forward to save space.

Next, we see details about the request we made. The full URL was http://127.0.0.1:8080/api/tutorial/1.0/employees. As we’ll see, the URL is an essential part of a REST API.

Then we see the request headers and the response. The response was empty. If this API was more user-friendly, it might return the employee id instead of making us specify it.

Finally, we see the important part. The response code was 201, which corresponds to Created. The request succeeded!

Let’s go back to the request for all records. Click on the blue GET box on the top again. Then, click on the Try it out! button.

The response body for this request is the list of employee records the server currently has. Click in the box and you can scroll.

Here are the results:

The last record in the list is the new one we added.

Failure

Let’s try to add another employee.

Enter this in the value box:

Then click Try it out! again.

Our response code this time was 403, which corresponds to Forbidden. We can’t add an employee with the same id number.

Here again, a more friendly API might generate ids for us, but this example illustrates how a service enforces data integrity via response codes.

REST URLs

Now click on the POST box again to collapse it and click on the second blue one that says GET.

REST API Tutorial Retrieve

This is the API method for retrieving individual employee records.

First, let’s take a look at that URL: /api/tutorial/1.0/employees/{id}

It has the same base URL as the one for creating employees. All of the URLs in the API contain GET /api/tutorial/1.0/employees.

HTTP methods are what define operations in well-formed REST APIs, not the URLs. URLs should not contain verbs.

The difference now is that the id of the employee is part of the URL.

You can think of the URLs as forming an index for the records.

There are four records in the server right now:

  • /api/tutorial/1.0/employees/1
  • /api/tutorial/1.0/employees/2
  • /api/tutorial/1.0/employees/3
  • /api/tutorial/1.0/employees/99

So when we want to retrieve, modify, or delete a record, we operate on its URL using the correct HTTP method.

If we want to retrieve all records or add to the set, we operate on the top-level URL: /api/tutorial/1.0/employees.

REST URLs usually include versions. I’ve set up this API to be version 1.0: /api/tutorial/1.0/employees

This convention provides a migration path for applications. A server can offer two different API versions via different URLs. Client applications can migrate to the new API incrementally or all at once.

Retrieving Data

Now let’s request an employee. Enter 99 in the value box and click the Try it out! button.

We get back our new employee and a successful response code.

Let’s try an invalid id.

So enter 122 and click the button.

Now we get an error document, and the response code is 500?

500 means Internal Server Error. The error document shows that our server threw a NullPointerException. The service has a bug!

Well-designed REST APIs will handle errors gracefully. In this case, we received a code that makes it clear there is a problem. But if we try another request, the service will respond.

Separation Between Client and Server

We’ve been exchanging small JSON documents with our server.

Let’s change the contents a bit.

First, go back to the POST area.

Then, enter this in the value text area:

And submit it.

It worked! The server will accept employees with no phone number.

Scroll back up to the first GET box where we can retrieve a list of employees and click the Try it out! button.

The result includes the new record.

An important aspect of REST is the separation between client and server.

In CRUD, and therefore REST, the implementation of the client and server are independent. The data forms the demarcation point between the two applications.

In this example, we demonstrated that the server is forgiving when it comes to required and non-required fields in employee records. This is an example of loosely-coupled design.

But we can go further with the separation between client and server. I wrote this example server in Java. It could be replaced with one written in C#, Python, or Ruby. If the URLs and the document remain the same, the client doesn’t have to change.

Delete

Let’s remove our incomplete record. Close the POST box and open the DELETE area under it.

The DELETE API method looks a great deal like the GET method. The URL is the same, complete with the employee id. Only the HTTP operation differs.

REST API Tutorial Delete

Enter 122 in the value text area and make a request. You should receive a 200 response code for success.

Scroll back up the first GET box. Then click the Try it out! button.

The result is the same list of employees we saw earlier.

Mr. Stark has left the building.

Now go back to the DELETE request area and try an invalid request. Enter 123 for the id.

REST API Tutorial Invalid delete

Response code 204 means no content. Nothing was deleted because we used an id that doesn’t exist.

Stateless

REST APIs are stateless. The server does not need to know about the state of the client and vice versa.

The state of the data is transferred between the applications, just as the name indicates. We added an employee record. We deleted the record.

This decoupling between components has distinct advantages. Many modern web applications, including those on phones, use REST. The expense of keeping a connection open and synchronizing state between client and server is too great, especially for an application with thousands of users.

The disadvantage is that when you design your application, you have to be mindful of the possible synchronization issues between client and server.

Updating Records

Let’s add Anthony Stark back to the server again.

Now go back to POST and add this record.

The response code will be 201. So if you go and list all employees, the new record is there.

We’ll want to add a phone number for this record.

Got to the brown PUT option on the bottom. PUT requires two values, the id of the record to be updated and a document with the new values.

Enter 122 for the id.

Enter a complete record for the employee.

REST API Tutorial First Update

Click the button. The response code is 200.

If we list our records again, we see the phone number.

Since the id of the record to be updated is a parameter on the path, we can change a record’s id.

So let’s move Mr. Stark to record 100.

REST API Tutorial Second update

Next, click the button. The response code is 200.

Then, if we list our records again, we see the new id.

Now go back to the PUT area and enter the same request without any changes.

The response code is 204. We tried to modify record 122 again, and it no longer exists.

Patching Records

Submitting partial updates to records is a relatively new operation and is not supported in all APIs.

Some APIs support partial updates via the PUT method, but PATCH is the more technically correct method for partial updates.

So let’s add another incomplete record and then PATCH it.

Add this employee:

Then, use GET to retrieve employee id 123.

We have nulls for both the last name and phone. Let’s add this employee’s last name.

First, click on the PATCH box.

REST API Tutorial Patch option

The options are identical to a PUT, but we can submit partial records.

Next, enter a record with only a last name. Then, add the employee id of 123.

REST API Tutorial Patch Blake

Finally, click the submit button. We will then receive a successful response code.

So let’s retrieve the record again.

We can update a record by only specifying the fields we wish to add with a PATCH.

Response Codes

As you can see, it’s critical for client applications to handle HTTP status codes correctly.

Response codes contain three digits. The HTTP standard divides them into ranges.

Codes that begin with 2 indicate success.

A code beginning with 4 means a client error.

A server error begins with 5.

Response Code Description
200 Success
201 Created
204 No content
400 Bad request
403 Forbidden
404 Not found
500 Internal server error

The Created code is returned for a new record, while Success is returned for modifying, deleting, and retrieving records.

The Forbidden response code indicates a record can’t be created, while No content indicates a failed modification.

Well-architected APIs will document which code they return under each possible condition.

Conclusion

We’ve used a simple API to examine how REST is used to exchange object state between a client and a server.

While we were adding, removing, updating, and retrieving records, we also looked at how APIs use HTTP methods and response codes to differentiate operations and results.

REST is the lingua franca of the modern web. It’s a protocol used for browsers, phones, and IOT devices. Now that you have a basic understanding of how to use it, you’re ready to build the next big thing.

 

image credit: dorota dylka

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑