Add GAE and GKE fluentd Handlers by waprin · Pull Request #2668 · googleapis/google-cloud-python · GitHub

On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations.

This adds two new handlers and some helper functions to detect when they should be used.

@dhermes /cc @jonparrott

Pretty awesome stuff!

Think I addressed all comments, let me make sure Travis is passing and then I'll do another once-over before I request another review.

@waprin Pretty please squash (either in the GitHub UI or in your local copy).

@jonparrott Do you want to do a final pass?

On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations. This adds two new handlers and some helper functions to detect when they should be used.

Add GAE and GKE fluentd Handlers

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 3, 2016

daspecster reviewed Nov 3, 2016

View reviewed changes

theacodes suggested changes Nov 3, 2016

View reviewed changes

theacodes self-assigned this Nov 3, 2016

dhermes reviewed Nov 3, 2016

View reviewed changes

This was referenced Nov 3, 2016

Python Logging API always defaults to Custom logs #2673

Closed

Convert logging doctest examples to testable snippets. #2675

Merged

dhermes reviewed Nov 4, 2016

View reviewed changes

waprin force-pushed the newhandlers branch from a81e8dd to 31641bd Compare November 4, 2016 20:56

dhermes reviewed Nov 4, 2016

View reviewed changes

waprin force-pushed the newhandlers branch 4 times, most recently from c755a71 to c132584 Compare November 16, 2016 21:50

dhermes reviewed Nov 18, 2016

View reviewed changes

Comment thread logging/unit_tests/test_client.py Outdated

self._scopes = scope

return self

def authorize(self, _):

This comment was marked as spam.

Sign in to view

dhermes reviewed Nov 21, 2016

View reviewed changes

dhermes approved these changes Nov 21, 2016

View reviewed changes

waprin force-pushed the newhandlers branch from 4477d59 to a65ef9f Compare November 21, 2016 22:21

waprin force-pushed the newhandlers branch from a65ef9f to 02ae299 Compare November 21, 2016 22:24

theacodes approved these changes Nov 21, 2016

View reviewed changes

dhermes merged commit 5d59291 into googleapis:master Nov 22, 2016

daspecster added the api: logging Issues related to the Cloud Logging API. label Nov 22, 2016

dhermes mentioned this pull request Dec 9, 2016

Update versions for mega-release. #2846

Merged

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017

Merge pull request googleapis#2668 from waprin/newhandlers

a787604

Add GAE and GKE fluentd Handlers

parthea pushed a commit that referenced this pull request Nov 24, 2025

Merge pull request #2668 from waprin/newhandlers

392f14c

Add GAE and GKE fluentd Handlers

-              Logging client.
+              It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+              There are different handler options to accomplish this. To automatically pick the default,
+              use :meth:``google.cloud.logging.client.get_default_handler`.

-              method :meth:`setup_logging <google.cloud.logging.handlers.setup_logging>` is provided to configure
-              this automatically:
+              CloudLoggingHandler
+              =======================

-                  :param excluded_loggers: The loggers to not attach the handler to. This
-                                           will always include the loggers in the path of
-                                           the logging client itself.
+                  :param excluded_loggers: (Optional)The loggers to not attach the handler

-                              target=self._run,
-                              name='google.cloud.logging.handlers.transport.Worker')
+                          thread_name = 'google.cloud.logging.handlers.transport.Worker'
+                          self._thread = threading.Thread(target=self._run,

+                      LOGNAME = 'loggername'
+                      MESSAGE = 'hello world'
+                      record = _Record(LOGNAME, logging.INFO, MESSAGE)
+                      expected_payload = {'message': MESSAGE,

		@@ -0,0 +1,6 @@
		Google App Engine Flexible Log Handler
		=======================================

		@@ -0,0 +1,6 @@
		Google Container Engine Log Handler
		=======================================

                   >>> cloud_logger.error('bad news')
+              It is also possible to attach the handler to the root Python logger, so that for example a plain
+              `logging.warn` call would be sent to Cloud Logging, as well as any other loggers created. However,

               """Python :mod:`logging` handlers for Google Cloud Logging."""
               from google.cloud.logging.handlers.handlers import CloudLoggingHandler
+              from google.cloud.logging.handlers.GAEHandler import GAEHandler

+                                           to. This will always include the loggers in the
+                                           path of the logging client itself.
+                  :type log_level: :mod:`logging` log level

               class _Client(object):
-                  def __init__(self, project):
+                  def __init__(self, project, _=None, __=None):

                       return self
+                  def authorize(self, http):
+                      pass

+              Logs to the well-known file that the fluentd sidecar container on App Engine
+              Flexible is configured to read from and send to Stackdriver Logging.
+              This file is largely copied from:

+              import math
+              import os
+              LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'

		LOG_FILE_COUNT = 3


		class GAEHandler(logging.handlers.RotatingFileHandler):

		self.assertEqual(payload, json.dumps(expected_payload))


		class _Record(object):


		client = self._makeOne(self.PROJECT, credentials=_Credentials())

		with _Monkey(os, getenv=lambda var: var == 'GAE_APPENGINE_HOSTNAME'):


		self.assertIsInstance(handler, GKEHandler)

		with _Monkey(os, getenv=lambda _: None):


		self.assertIsInstance(handler, GAEHandler)

		with _Monkey(os, getenv=lambda var: var == 'KUBERNETES_SERVICE'):



		class GAEHandler(logging.handlers.RotatingFileHandler):
		"""A handler that writes to the GAE fluentd Stackdriver logging file.

+              class GAEHandler(logging.handlers.RotatingFileHandler):
+                  """A handler that writes to the GAE fluentd Stackdriver logging file.
+                  Writes to the file that the fluentd agent on App Engine Flexbile is

+                      import logging
+                      import json
+                      handler = self._makeOne()
+                      LOGNAME = 'loggername'

                 logging-sink
                 logging-handlers
+                logging-handlers-GAEHandler
+                logging-handlers-GKEHandler

+              Google App Engine Flexible Log Handler
+              =======================================
+              .. automodule:: google.cloud.logging.handlers.GAEHandler

+              Google Container Engine Log Handler
+              =======================================
+              .. automodule:: google.cloud.logging.handlers.GKEHandler

+                  def test_setup_logging(self):
+                      from google.cloud._testing import _Monkey
+                      import google.cloud.logging.client as MUT
+                      client = self._makeOne(

+                      self.called = False
+                  def __call__(self, _, log_level,
+                               excluded_loggers=None):  # pylintdisable=unused-variable

               Integration with Python logging module
-              ---------------------------------------------
+              ---------------------------------------

                                              credentials=_Credentials(),
                                              use_gax=False)
-                      os.environ[_CONTAINER_ENGINE_ENV] = True
+                      os.environ[_CONTAINER_ENGINE_ENV] = "True"

Add GAE and GKE fluentd Handlers by waprin · Pull Request #2668 · googleapis/google-cloud-python · GitHub

Uh oh!

Add GAE and GKE fluentd Handlers - #2668

Add GAE and GKE fluentd Handlers#2668dhermes merged 1 commit intogoogleapis:mastergoogleapis/google-cloud-python:masterfrom waprin:newhandlerswaprin/google-cloud-python:newhandlersCopy head branch name to clipboard

Conversation

waprin commented Nov 3, 2016

Uh oh!

waprin commented Nov 3, 2016

Uh oh!

daspecster left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

Add GAE and GKE fluentd Handlers#2668
dhermes merged 1 commit into
googleapis:masterfrom
waprin:newhandlers

+              (i.e. your app.yaml contains ``runtime: python``), and
+              :class:`~google.cloud.handlers.container_engine.ContainerEngineHandler`,
+              which is
+              recommended when running on `Google Container Engine` with the Stackdriver

+              from google.cloud.logging.handlers import AppEngineHandler
+              from google.cloud.logging.handlers import ContainerEngineHandler
+              from google.cloud.logging.handlers import setup_logging
+              from google.cloud.logging.handlers.handlers import EXCLUDED_LOGGER_DEFAULTS

+              """
+              # This file is largely copied from:
+              # https://github.com/GoogleCloudPlatform/python-compat-runtime/blob/master/appengine-vmruntime/vmruntime/cloud_logging.py

-                 >>> setup_logging(handler, excluded_loggers=('werkzeug',)))
+                  >>> handler = CloudLoggingHandler(client, name="mycustomlog")
+              fluentd logging handlers

+              the environment to automatically detect whether the code is running in
+              these platforms and use the appropriate handler.
+              In both cases, the fluentd agent is configured to automatically parse log files

+                          'message': message,
+                          'timestamp': {
+                              'seconds': 5,
+                              'nanos': int(.03 * 1e9)

+                                          'timestamp': {'seconds': 5,
+                                                        'nanos': int(.03 * 1e9)},
+                                          'thread': record.thread,
+                                          'severity': record.levelname}

+              class AppEngineHandler(logging.handlers.RotatingFileHandler):
+                  """A handler that writes to the App Engine fluentd Stackdriver log file.
+                  Writes to the file that the fluentd agent on App Engine Flexbile is

+                  itself, and not as a formatting step, so as not to interfere with
+                  user-defined logging formats.
+                  Logging calls can also alternatively 'trace_id' in as a field in the

+              """Logging handler for Google Container Engine (GKE).
+              Formats log messages in a JSON format, so that Kubernetes clusters with the
+              fluentd google cloud plugin installed can format their log messages so that