FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Modified Streaming and Generating Custom Search Command by akaila-splunk · Pull Request #407 · splunk/splunk-sdk-python · GitHub

Repository navigation

Modified Streaming and Generating Custom Search Command - #407

Merged
ashah-splunk merged 18 commits into
developfrom
DVPL-9943
Nov 3, 2021
Merged

ashah-splunk merged 18 commits into
developfrom
DVPL-9943

Conversation

Copy link
Copy Markdown
Contributor
  • Fix field dropping issue for Streaming and Generating CSC

fantavlik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This approach is really looking good, thanks for the documented examples and generating command test! Just one request to add a full fledged tests for the add_field scenario to ensure that is working as intended.

fantavlik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This approach is really looking good, thanks for the documented examples and generating command test! Just one request to add a full fledged tests for the add_field scenario to ensure that is working as intended.

Comment on lines +176 to +178
def add_field(self, current_record, field_name, field_value):
self._record_writer.custom_fields.add(field_name)
current_record[field_name] = field_value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This function should return current_record or else the documentation is incorrect.

Suggested change
def add_field(self, current_record, field_name, field_value):
self._record_writer.custom_fields.add(field_name)
current_record[field_name] = field_value
def add_field(self, current_record, field_name, field_value):
self._record_writer.custom_fields.add(field_name)
current_record[field_name] = field_value
return current_record

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Good catch @Bre77 - I think in this case the documentation needs to be updated, the current_record should be updated with the new field without the need to return since it's essentially pass-by-reference for this usage (I always go back to this article when I need to refresh my memory: https://robertheaton.com/2014/02/09/pythons-pass-by-object-reference-as-explained-by-philip-k-dick/).

So if you all agree can we fix the README.md above to the following?

class CustomStreamingCommand(StreamingCommand):
    def stream(self, records):
        for index, record in enumerate(records):
            if index % 1 == 0:
                self.add_field(record, "odd_record", "true")
            yield record

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

And to make sure that example we provide is working ^ that's probably a good implementation to add when adding tests for the StreamingCommand / add_field case

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I've changed my implementation to using pass by reference (as per the suggested doco update) and everything works as expected.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Good catch @Bre77
I've updated the README accordingly.

output_records = [i for i in output_iter.next().data]

# Assert that count of records having "odd_field" is 0
assert len(list(filter(lambda r: "odd_field" in r, output_records))) == 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Really nice demonstration of the issue!

fantavlik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Looking forward to getting this out to our eager users 🚀 nice work

ashah-splunk merged commit e6caf2c into develop Nov 3, 2021
vmalaviya-splunk mentioned this pull request Nov 11, 2021
akaila-splunk deleted the DVPL-9943 branch May 2, 2022 10:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants


Back | FazBrowse Home | New Git URL