tl;dr thanks to a large Greek man, I know a surprising amount about a centralized app logging service known as fluentd
My mentor this summer is a 6 foot buff Greek man who goes by soto. To put it bluntly, he both terrifies me and I worship him.
He's really smart, and he's got this air about him that screams a mixture of "yolo" and "work is life". It's basically "work hard, play hard," but for a 31-year-old greek man.
His teaching style is very hands-off, and I have to admit I kind of like it that way. He wants me to figure shit out for myself, and so do I. He told me from the start that he hates when people ask him questions twice, so I don't.
For a while I followed him around and did simple tasks for him, but recently he told me to try to figure out fluentd, a centralized app logging service hosted by treasure data. Apparantly Logstasher, the service we currently use, crashes a lot, and so he wanted the switch.
So there I was, an 18-year-old helpless intern plunged into a task I knew nothing about. (I didn't even know how rails logging worked). I learned later that he told the development team that this was an "impossible task". Well, I showed him ;)
So now, two weeks (and some sweat and tears) later, I'd like to share everything I learned about logging in rails and sinatra applications, how to convert their default ruby loggers into fluentd loggers, and how to forward logs to a remote fluentd logserver. Hang on tight folks, cause it's gonna be a lot. And maybe if some other poor intern gets thrown into the same task I was, they'll find this helpful. :)
To start, let's go back to basics. How does rails logging work? By default, rails will log to log/{environent}.log. So when you're in the development environment, visiting localhost:3000, logs will all go to the file log/development.log.
To create custom logs in a rails app, you can easily tell Rails to use a different logger, like this:
Rails.logger = Logger.new("#{Rails.root}/log/user_logins.log")
When you want to log something to that logger, you can then call it:
Rails.logger.info "#{@user.id} logged in at #{Time.now}"
If you want to use the default logger (like log/development.log) but you want to log something extra to it, you can just call the logger from anywhere in your application:
logger.info "Cool log my friend"
To send your basic rails logs (log/development.log) to a fluentd server is actually pretty straight-forward, but we'll take this step-by-step. Using the gem act-fluent-logger-rails, you'll send over your logs to a specified port and IP address. Configured on that server will be a running fluentd service, which will accept connections from that specified port. Fluentd calls these "inputs". Each log sent over to fluentd will be tagged with a string of your choosing, and the fluentd service will match "tags" and then "output" those logs to an output of your choosing. You can be fun and use ElasticSearch + Kibana to see a pretty visual representation of your logs, or you can be boring and output to file, which just means writing those logs to a specific file of your choosing. Since we wanted our logs to be hosted on a different server, we needed to encrypt those logs before they were forwarded over, which fluentd also does. Now let's start with basic logging.
If you just want to use the basic logs (i.e. log/development.log or log/production.log, etc.) act-fluent-logger-rails is actually a great gem that will get the job done easily. Fluentd has a great mini-tutorial on using the gem, and I highly recommend you check it out (you'll only need the first four steps to get the gist of it). All you'll need to do is add a file to your config folder and tell Rails which logger you want to use in the separate config/environments/ files. However, there are two big flaws in the act-fluent-logger-rails gem. The first is that it is only capable of logging strings and (because of a pull request I made :P) exceptions. So if you want to log a hash, or an array or something, it will break. The second is that it doesn't let you customize your loggers with different tags. You get one logger (and one tag per environment) for everything.
The easiest way to customize your logging is to use fluent-logger-ruby, and define a different logger in the areas where you'd want to log with a different tag, like this:
1
2
3
4
require 'fluent-logger'
log = Fluent::Logger::FluentLogger.new('MyApp', host: 'localhost', port: 24224)
log.post("user_log", {"Login" => "Success" })
However, the problem I found was that a lot of our code had already been written for the Ruby Logger, and it would often call "info", "debug", "error", etc., on the logger, and so replacing the logger variable with a fluentd logger wasn't enough. In addition, things in our code like Delayed Job expected to be able to call those kinds of methods ("info", "debug") on the logger it was provided, but all logging was done in the background. So, I created a gem! Check it out! (shameless plug). It's called fluent-logger-sinatra just because I first made it for Sinatra apps, but the premise is the same. If you have a lot of pre-written code you don't want to change, or if perhaps other APIs are expecting a regular Ruby logger, this gem lets you just change one line and get all logs sent to fluentd. Speaking of Sinatra...
Sinatra is the main reason I created the gem, and that's because Sinatra expects to be able to call "write" on the logger it is provided, and it does all it's logging in the background. Normally, you would provide Sinatra with a basic ruby logger, and add in the "write" method like this:
1
2
3
4
5
6
7
require 'logger'
Logger.class_eval { alias :write :'<<' }
logger = ::Logger.new(::File.new("log/app.log","a+")
configure do
use Rack::CommonLogger, logger
end
With fluentd and the fluent-logger-sinatra gem, you can just add this to your code:
1
2
3
4
5
6
require 'fluent-logger-sinatra'
logger = FluentLoggerSinatra::Logger.new('MyApp', 'logs', '127.0.0.1', 24224)
configure do
use Rack::CommonLogger, logger
end
All logs will be tagged with "MyApp.logs", and you can match them from there. As I mentioned before, FluentLoggerSinatra also works great with Rails apps :)
Now that we know how to send logs to a fluentd server, it's time to learn how to recieve them in Part 2