This post discusses how easy it is to manage your log when you use containers. You can easily use Docker to redirect your logs to a single file. How cool is that? I was in the midst of a big deployment to AWS when I wrote this one. Docker is the only way to go.

The original post is over on Scalyr’s blog.

---

Sometimes, when troubleshooting or monitoring a Docker container, we need to see the application’s output streams. Containerized applications generate standard output (stdout) and standard error (stderr) like any other software. The Docker daemon merges these streams and directs them to one of several locations, depending on which logging driver is installed. The default driver makes the container output easy to access, and if we want to copy the information to another location, we can redirect docker logs to a file.

Let’s take a look at how we can manipulate logs generated by the default json-file log driver. This driver places the log output in a system directory in JSON formatted files but provides a command line tool for displaying log contents in its original format.

Viewing Container Logs

Let’s start with a simple container. We’ll create one that prints a message every second.

This command is an example from the Docker website. It executes a minimal Docker container with a Bourne shell script that logs a timestamp once every second. We’ll use it for a couple of examples.

So the easiest way to view container logs is with the docker logs command. Start the container and then display the logs.

This command displays all of the output the container has generated so far.

If there’s too much output for a single screen, pipe it to a buffering command like less.

This will still send all of the output generated so far through less, so you’ll need to page through to get to the end.

But sometimes you want to follow the logs as the container is running. There’s a command line option with an appropriate name for that: run docker logs with –follow.

After printing all of the container output so far, the command will continue to follow the logs. You can use the short form of the option -f, similar to the tail command, too.

The docker logs command also offers an array of convenient options, such as filtering output based on time and displaying timestamps. You can read more about them here. But what if you want to redirect the docker logs to a file for viewing offline?

Redirect Docker Logs to File

Since Docker merges stdout and stderr for us, we can treat the log output like any other shell stream. To redirect the current logs to a file, use a redirection operator.

To send the current logs and then any updates that follow, use –follow with the redirection operator.

This command saves the output for future reference, but what if you want to keep the logs and view them at the same time? Opening another shell and tailing the output file feels like a hack. There must be a better way.

So, let’s use tee to watch the output and save it to a file at the same time.

We still see the container output. Now, open another shell and check the output file.

The file is being updated too! We have the output on the terminal, and tee saves it to the file we specified at the same time.

Redirecting Stdout/Stderr in the Container

Of course, there’s another way to save your container logs to a file. Instead of sending output to stderr and stdout, redirect your application’s output to a file and map the file to permanent storage outside of the container.

When to eschew the standard output completely is a judgment call that you should make when you take into account how much output your application generates, how often you think you’ll need to refer to it, and what kind of logging infrastructure is available to you. If, for example, you need to store events for compliance purposes, a logging framework like Java’s Logback may be a better option than capturing stdout and stderr.

That said, let’s modify the previous example to save the output to a file. We’ll change the loop to print the date to a file in /tmp. We’ll also run the container with /tmp mapped to the same directory of our host. Note that Docker will not accept relative paths on the command line, so if you want to use a different directory, you’ll need to use the complete path.

Take a look at /tmp/output.log on the host system.

Now, take a look at docker logs.

There’s nothing there. Since we redirected the output to a file, there is nothing for Docker to capture. Of course, we would need to capture both stdout and stderr in a production application.

Use the Method That Works for You

We covered several different ways to capture docker logs for saving and analysis. The best method is the one that works for you. In a development environment, the docker logs command is a powerful tool that works well with other command line tools. You can use Docker’s built-in tools to view, filter, and copy logs to a file.

In a production environment, log management should be part of an enterprise strategy. Scalyr’s log aggregation tools help you aggregate, process, search, and analyze your logs. Contact us, or sign up for a free trial to see how our tools make it easier to find what you need in the vast ocean of log entries produced by your growing application infrastructure.

 

 

Why is unit testing important? This is one of those titles that you almost have to click. I wrote this one for NCrunch, a testing tools company, a while back. The original is here.

 

---

What’s the point of unit testing? Why is it important? Even if you’ve never heard the question asked in so many words, you’ve probably been in a situation where your team ignored unit tests or treated them as an afterthought. They’re often the first thing to go when time runs short.

So, is unit testing important? How critical can it be if so many teams think it’s okay to ignore it? Well, the fact is unit testing is crucial to a development team’s success. Neglecting it can result in lost time, wasted effort, and missed opportunities.

We already debunked the most common excuses for not using tests and TDD. We know that we should write unit tests. Just like we know we should brush our teeth after every meal, exercise every day, and spend less time on social media. But is unit testing important?

Let’s take a look at why unit testing isn’t just important but is also critical to development success.

Unit Tests Are Documentation

For most developers, the fastest way to learn how to use new code is by reading an example. Documentation is useful, but nothing beats an example. This is especially true for a unit test, which is sample code that you can run in your IDE. What’s the first thing you scan for when you’re reading documentation? I bet it’s code snippets.

What’s the first thing you look for when you browse through the source code? Tests. What could be more useful than a collection of working unit tests? They’re examples of the code in use. They’re easy to run, and you can copy or change them for experimentation.

Is there anything more frightening than source code that doesn’t come with any tests at all? If you’re inheriting it from another developer, having no unit tests is downright terrifying. Production code and sample programs often have external dependencies and may not run in a development environment. Even if you can run them outside of production, experimenting with them can be difficult too. You can run unit tests in place, in your development environment, or on the command line, and they are easy to play with.

Unit Tests Catch Regressions

How do you know when a code fix or new feature doesn’t break existing behavior? Hopefully, your answer is unit tests. Useful unit tests establish baseline behavior for your code in isolation. By running them as part of each build, you catch unforeseen side effects before you release the code into the wild.

Detecting code regressions early in the development process is the most recognized practical application of unit tests. But the benefits of testing go beyond catching problems early; tests help you isolate problems quickly, too. By providing feedback right away, they make it easier to correlate changes to bugs.

Unit Tests Provide Immediate Feedback

How long do you want to wait before you see your code work? After you deploy it to QA? With unit tests, you don’t have to wait at all. Feedback is precisely one build away. It’s impossible to overstate the importance of the tight feedback loop unit testing offers developers.

As I mentioned above, immediate feedback simplifies troubleshooting. Rather than trying to find a new regression after the fact via log messages or a description of the problem, bugs surface immediately and can be correlated directly to the last code change. If you’re already accustomed to running frequent builds, it’s possible to isolate a new issue in minutes.

But this immediate feedback isn’t just for isolating problems. It’s also useful for confirming new behavior too. Developers aren’t lawyers, and we spend a lot of time asking questions we don’t already know the answers to. Unit tests give us the rapid responses we need. This tight feedback process saves time and improves our code.

Unit Tests Improve Code Quality

What’s the best way to figure out how to design your code? By using it.

Translating use cases and requirements into software is only the first step in the development process. It’s only after writing tests that you can use the new code to solve problems so you know you have a usable product. (Of course, inverting the process by writing the tests first is even better.) It’s not uncommon to find that an interface or object is hard to use or that you missed a set of features entirely in your first pass at developing a product. Unit tests shorten this discovery process. Designing and implementing tests makes you your own first customer. In a way, unit tests create a second type of feedback: how does this code look? Is it easy to use? Does it solve the problems it’s supposed to solve?

Depending on how you work, you may write your tests after you’ve implemented your code. After you think you are feature complete or at some other natural break point, you create tests that verify critical aspects of the code’s behavior. You might decide that something is shaped wrong, and you may change a method signature or some names to make things more transparent to a user.

Or, you might start with a test that defines what you want the code to do, and then write code that passes the test. Yes, I am referring to test-driven development (TDD.) Kent Beck says that TDD “gives you a chance to learn all of the lessons that the code has to teach you.” When you put tests up front, you write software that you understand better. By virtue of writing tests up front, you’ve already used your own software once.

Unit Tests Build Confidence

There’s a common thread to this list of reasons why unit tests are important. In each case, units tests build confidence about your code. Unit tests ask questions, and the answers provide assurances about the reliability and quality of your software throughout its lifecycle.

Code that includes unit tests comes with an implied guarantee. You can rely on the behavior they verify. If you make changes to the code and the tests still pass, you can be confident that you haven’t broken anything.

Developing tests as you write your application code builds confidence too. Whether you’re using TDD or not (and you should), unit tests are a stake in the ground for how the software behaves. The immediate feedback that your unit tests provide reassures you that you are headed in the right direction. Checking in code with working tests makes the rest of your team feel better too.

When it comes time to take your code off the shelf and update, repair, or change it, a test suite offers confidence too. You can pick up where the last developer left off, in a tight code-test-repeat development cycle. You can catch new regressions as you work and add tests to an existing set of verifications.

So, why is unit testing important? Think about what you give up without them. Without tests, you don’t get that tight feedback loop as you work. You have to wait for QA and integration testing to see your application in action. Simply put, if you don’t develop with unit tests, you can’t be completely confident that your code is ready for release.

 

 

Scalyr’s “Getting Started” blog series has covered a ton of different platforms and languages. I’ve had the privilege of writing a handful of the posts, and each one of them has been a lot of fun. This one is about logging with Apple’s Swift language. You can find the original post here.

---

We’ve covered how to log in seven different languages so far: C#, Java, Python, Ruby, Go, JavaScript, and PHP. We’ve also included a few libraries and platforms, like Log4J, Node.js, Spring Boot, and Rails.

Now, it’s time to talk about Apple’s Swift language. Swift has been slowly gaining in popularity, especially with since its open source release.

I’ll start with a quick example of manual logging in Swift. Then I’ll discuss details of how and why logging matters. Finally, I’ll move on to using Apple’s Unified Logger in a Swift application and how it can improve your ability to monitor applications and track down issues

The code example will be for MacOS, but you can easily adapt it for any Apple platform.

Let’s get to work!

The Simplest Swift Logging That Could Possibly Work

Let’s start by creating a new project in Xcode. If you don’t already have it installed, go to the App Store and install it from there.

Next, start Xcode and select Create a new Xcode Project from the first dialog window.

We’re going to keep the code as simple as we can in this tutorial, so select MacOS and Command Line Tool and then click Next.

Now, you can give your project a name. These values are not critical, but make sure that Swift is set as the project language.

Finally, we have a project.

Select main.swift in IDE, and you can see the code Xcode added for us.

Rather than print to the console, we want to save logging information to a text file.

Swift works differently from many of the other languages we’ve worked with before. There are several ways to write a string to a file. But since the purpose here is to demonstrate the easiest way to, here it is.

Set the file name to somewhere you can quickly locate it. Keep in mind that the working directory for Xcode projects is buried somewhere deep in your Library directory, so you may want to set the log file location to your home directory.

When you’ve picked a place for the file, build and run the project. Then find the file and open it with any editor you wish. You’ll see your log message.

We started with declaring where our log file will be. As we’ll see below, this is important and something all components have to know—unless there’s a framework that manages log locations for us.

Next, we converted that file name to a URL. All Swift strings can write themselves to a URL, so we’re taking advantage of that.

Finally, we created a string and wrote it to the URL.

While this would work for logging simply, it wouldn’t necessarily work well. What’s the cost of having every log message open a URL and write itself to it? I doubt it scales well for hundreds or thousands of log messages. What would happen when more than one message wanted to log itself at the same time? Is this mechanism threadsafe? Would it block the entire program?

Let’s look at a better way.

What Is Application Logging?

First, let’s take a look at what logging is.

You’ve probably dealt with logging before and have an idea of what it means, but it’s still worth defining it so we can evaluate a logging framework in terms of what we need. Back in the first article in this series, we defined application logging like this:

Application logging involves recording information about your application’s runtime behavior to a more persistent medium.

Let’s break that down.

Logging means recording runtime behavior. We’re documenting events about an application. The time factor implies in chronological order.

Second, we store logs in a more persistent medium. Application events occur quickly, and we want to be able to go back and review them. Maybe we’re trying to track down an error, or perhaps we have to maintain an audit trail. A persistent medium can be a disk, a relational database, or a search engine.

So, that’s what logging is. How can you do it more effectively in Swift, and why would you want to? These are important questions. Let’s get back to work,

Apple’s Unified Logging System

With MacOS Sierra and iOS 10, Apple introduced the new unified logging system. While they haven’t deprecated the legacy logging system yet, the new logger is the direction they’re going in.

The unified logger has built-in security and performance features that make it desirable, even though it does have an idiosyncrasy that many developers dislike: messages are stored in memory and a proprietary data store. There are no clear-text log files, and messages can only be read with Console.app or the command line log tool.

Let’s add the unified logger to our application and see how it works.

Let’s start by removing the code for manipulating files and just replacing it with os_log(), the unified logger. We need to import the os package, and add the call to os_log.

This program doesn’t build.

We can only pass static strings to os_log. So, we’ll change how our string is defined.

Build and run this version, and everything looks fine.

But where’s the log message?

Run the Console application. You can find it in Applications/Utilities or by opening Spotlight and searching for it.

Search for the text Process: Swift in the search bar as shown below. (Click on the image for a larger version.)

There’s our log message.

The unified logger takes care of the log file location by storing all logs in the same system and indexing them. We don’t have to worry about where they belong, how to open them, and when to close them.

So what do we need to concern ourselves with when it comes to Swift logging?

Why Log?

Someday you’ll write that perfect application. You know which one—the one that has no bugs, reacts correctly to unexpected events, and anticipates your customers’ every need. But until then, you’ll need a way to keep an eye on things. Here in the real world, even well-constructed applications have bugs and other shortcomings, and we need to monitor what’s going on inside them.

We can find and fix problems quickly when we’re developing code. We have the debugger, we can add print statements, and yes, we even use logs as we’re coding. We’re watching our application in captivity, in a sterile environment where we have control.

But our application is going to leave the tender embrace of our development environment eventually. It will be used in ways we didn’t anticipate, and it will encounter conditions we never imagined. And then, when it fails, we’ll need a way to figure out what happened.

This is where logs come in.

Isolating bugs isn’t the only use for logs, either. They’re useful for watching an application that works and finding opportunities for improvement. We can use them to detect patterns of behavior that can be improved upon. We might be able to discover a resource that should be cached in memory instead of a file or a feature that’s never used.

And, of course, the usefulness of logs doesn’t stop there. Sometimes we need an “extra copy” of transactional data, such as credit card charges or other financial transactions.

We need visibility into our application’s runtime behavior, and we get it with logging.

How Should You Log?

We took a quick look at Apple’s Unified Logging System. It fulfills the requirement for persistently storing logs in a central location. We can find them with a text search via the console application or a command line tool.

But it would be nice if the logs were easier to find inside the system and had a bit more context.

This leads to a more general question: what information should be logged?

Most logging systems include at least the following information in each log entry:

• Timestamp: when the event described in the log entry happened. The unified logger takes care of this for us.
• Event context: useful information about the event. “It worked” or “It’s broken” might be useful or entertaining during a debugging session. “Failed to connect to database at 192.9.168.3:5000” is more useful in production.
• Severity level: logs tend to have a level that puts them in context compared to other entries. The unified logger defines default, info, debug, error, and fault. We’ll take a closer look at logging levels below.

So the unified logger takes care of two out of three essential aspects of logging for us. But it’s up to us to manage the content of our messages effectively.

These are only the basics when it comes to logging best practices. Scalyr has an article about logging best practices here. There’s also an excellent cheat sheet here.

Customizing the Unified Logging System

Adding a Log Level

So, let’s make our use of the unified logger more effective. We can already log information to persistent storage with a single line of code. What more can we do?

First, let’s add some additional context to our logs with log levels. Change the call to os_log by adding a few more arguments, and the log message text.

Build and run and take a look at the console application.

The logger tagged the message as an error for us because we passed .error for the type: argument. We can also pass in .info, .debug, or .fatal. if we omit the type: argument, the message is logged as default.

To do this, we had to pass the call to os_log the log message, an instance of a logger, along with the log message type.

Let’s take a closer look at the notion of a log instance before continuing to customize our messages.

Log Instances

We’d been calling os_log with a single call to log messages up until the previous exercise. This is a convenience function. When we called it with only a message, it used a default logger instance. When we called it with an instance and a log type, we were calling a different function that manipulated the instance we passed in.

Let’s take a closer look at our log message in the console application.

Because we used the default logger instance, the message doesn’t have subsystem or category information.

If we want to populate these fields, we need our own logger instance—one that’s initialized with information about our application. Let’s create one.

The first step is to create a plist with logging configuration information.

In Xcode create a new file. Select the property list type.

Name it com.scalyr.SwiftLogger and save the file.

The file will open in the property editor. Life is too short for that. Right click on the file name in the project navigator, and select open as and then source code.

Enter this text.

This property list defines a logger system that defines how info level messages are handled. We’ll go over this after this example.

Copy this file over to the /Library area of your system

You’ll need to be in the working directory of your project and have root access.

This property list defines a logging subsystem named com.scalyr.SwiftLogger that we can refer to when we create a new instance.

So, let’s update the code.

We’re creating a logger instance with OSLog.init. We pass it the name of our subsystem and then a logging category. A category is an arbitrary name for group log messages.

Next, we pass the logger instance to os_log.

Run the code, and you’ll find a log message, set to the default level, in the console app. Click on it to see the details.

Let’s send this message as a fault.

Run this version, and check the console application.

We see the details and the fault log level.

Log Message Storage

In the plist, you can see this section toward the middle.

This defines the storage options for the info log level. It tells the system to inherit the default. We only needed to establish a log system to get past the step of creating a named logger.

But it’s useful to understand the default rules for storing logs.

Default log messages are stored in memory buffers. The system will compress and move them to the data store as memory buffers fill.

Info log messages are stored in memory buffers. The system won’t move them to the data store. It purges them as memory buffers fill.

Debug messages are only captured in memory when debug logging is enabled through a configuration change. The system will never write them to disk.

Unified logging always saves error and fatal messages to the data store.

Log Contents

We can set up a log with a subsystem and category, and we can control the level messages we send with it. That leaves log contents.

Even though we’re limited to passing static strings in the log message, we can still use C-style formatting with those strings.

So we can use log strings as a formatter.

Run this from Xcode, and you’ll see the log message.

We can log scalar values too.

But what we see from inside Xcode is not what happens in production.

Let’s change the code back to printing a string in the log message.

Run this in Xcode, and you’ll see “The answer is a secret” in the console application.

Now, let’s build the application and run it from the command line.

Open a shell and move to the project directory. Run xcodebuild and then run the application.

Check the log message in console.

The unified logger redacted the string. This is the default behavior for string messages. It isn’t for scalar values.

If you do not want a string to be redacted use the {public} notation.

If you want a scalar value redacted, use the {private} notation.

This is a built-in security feature that helps protect critical information. As tempting as it might be to mark everything public, resist the urge.

What Now?

We’ve covered the hows and the whys of logging with swift on an Apple platform with the unified logger. It’s a comprehensive system that makes logging easy for an application developer.

Even though we covered a lot of ground, there’s more to know about logging. Start here, and don’t forget the cheat sheet.

The Apple unified logger documentation is here for developers.

Scalyr’s log aggregation tools help you aggregate, process, search, and visualize your logs. This makes it easier to find what you need in the vast ocean of log entries produced by your growing applications.

So now that you know the fundamentals, get started with logging in your Swift applications today!

 

 

DZone has just released their 2019 Guide to Java and I am proud to say that I wrote an article for it! You can download the whole guide here, or check out my post over here.