DVPL-12514: Improve existing CSC app tests by cecyliaborek · Pull Request #633 · splunk/splunk-sdk-python · GitHub

Fixed a flaky test tests/searchcommands/test_csc_apps.py::TestCSC::test_eventing_app caused by:

Unreliable data source – The test relied on events from Splunk’s _internal index, which is not guaranteed to contain the required data.
Malformed search query - missing status field caused eventingcsc status=200 to raise KeyError, which wasn’t caught by the test assertions.

The Fix

Created a helper method _create_test_data_search that creates synthetic events for the eventingcsc command
Made assertions stricter to check both result count and field values.

For reasons yet unknown to me, we check for is_preview in results - here's how I dealt with this for Splunk 10. Maybe it'll be useful
c14f4f3

To be honest, I don't understand what is going on here. Why are we testing for errors in Splunk? I thought what actually needs fixing is the query. Otherwise, the test that was already there looks fine to me - it tests the eventing app's behaviour, not Splunk's.

To be honest, I don't understand what is going on here. Why are we testing for errors in Splunk? I thought what actually needs fixing is the query. Otherwise, the test that was already there looks fine to me - it tests the eventing app's behaviour, not Splunk's.

I don’t think the goal is to test the eventing app itself (it’s just a simple example) but rather the SDK components used to build it. My thinking was that it’s good to cover not just the happy path, but also cases where something is done incorrectly, and ensure the SDK handles that properly. That said, you’re right — maybe we should focus on testing cases where an app is created incorrectly, rather than cases where it’s just used incorrectly.

cecyliaborek force-pushed the dev/CB/DVPL-12497-fix-csc-apps-test branch from 5fc4ec1 to d514410 Compare August 11, 2025 09:33

Ickerday reviewed Aug 11, 2025

View reviewed changes