FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

python.d.plugin: use separate process for initial module checking by ilyam8 · Pull Request #5552 · netdata/netdata · GitHub

python.d.plugin: use separate process for initial module checking - #5552

Merged
ilyam8 merged 17 commits into
netdata:masterfrom
ilyam8:pythond_plugin_refactor
Mar 7, 2019
Merged

python.d.plugin: use separate process for initial module checking#5552
ilyam8 merged 17 commits into
netdata:masterfrom
ilyam8:pythond_plugin_refactor

Conversation

ilyam8 commented Mar 5, 2019
edited
Loading

Copy link
Copy Markdown
Member
Summary

This PR adds (major) changes only to python.d.plugin file.

Fixes: #5525

pyhton.d.plugin imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.


Memory utilization comparing before/after the PR (one job example module, py3.7.2):

21.1 => 8.8 MiB

Component Name

collectors/python.d.plugin

Additional Information

This PR adds separate process for initial module checking.

Logic:

  • main process spawns checker process
  • checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list.
  • checker process returns list of modules and jobs.
  • main process loads only active modules, etc.

netdatabot added the area/collectors Everything related to data collection label Mar 5, 2019
netdata deleted a comment Mar 5, 2019
netdata deleted a comment Mar 5, 2019
netdata deleted a comment Mar 5, 2019

ilyam8 commented Mar 5, 2019

Copy link
Copy Markdown
Member Author

ok, i tested it with on Manjaro (py 3.7.2) and on Centos6(py 2.6.6), and it appears to be working correctly

ilyam8 changed the title [wip] python.d.plugin: use separate process for initial module checking python.d.plugin: use separate process for initial module checking Mar 5, 2019

ilyam8 commented Mar 5, 2019

Copy link
Copy Markdown
Member Author

Hey @Ferroin if you could find a time and test the PR a bit it would be very nice

netdata deleted a comment Mar 6, 2019
netdata deleted a comment Mar 6, 2019

ilyam8 commented Mar 6, 2019
edited
Loading

Copy link
Copy Markdown
Member Author

ok, i don't think we need to run a job if job.check() returns False in main. It can be error prone, instead we can treat all jobs in main as autodetection jobs, so if a job's check(), for some reason, succeed in child process and failed in main we will retry check again and again

netdata deleted a comment Mar 6, 2019

ilyam8 commented Mar 6, 2019

Copy link
Copy Markdown
Member Author

@ktsaou if you are ok with the changes this is ready.

netdata deleted a comment Mar 6, 2019

ilyam8 commented Mar 6, 2019

Copy link
Copy Markdown
Member Author

@cakrit please approve

ilyam8 merged commit 2175673 into netdata:master Mar 7, 2019

ilyam8 commented Mar 7, 2019

Copy link
Copy Markdown
Member Author

ok, here we go, i am on a high alert ⏰

ilyam8 deleted the pythond_plugin_refactor branch March 12, 2019 13:48
jackyhuang85 pushed a commit to jackyhuang85/netdata that referenced this pull request Jan 1, 2020
…tdata#5552)

##### Summary

This PR adds (major) changes only to `python.d.plugin` file.

Fixes: netdata#5525

`pyhton.d.plugin` imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.

___
Memory utilization comparing before/after the PR (one job `example` module, py3.7.2):

> 21.1 => 8.8 MiB

![screenshot_20190305_111837](https://user-images.githubusercontent.com/22274335/53791147-c27a6e00-3f39-11e9-8eaf-8ac3809a3b6e.png)


##### Component Name
[`collectors/python.d.plugin`](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/python.d.plugin.in)

##### Additional Information

This PR adds separate process for initial module checking. 

Logic:
 - main process spawns checker process
 - checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list.
 - checker process returns list of modules and jobs.
 - main process loads only active modules, etc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/collectors Everything related to data collection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL