Issue Summary
Alerts are triggering but not firing off PagerDuty. The queries are triggered, but nothing fires to PagerDuty. I have tried the webhooks and those are firing but nothing on the Pagerduty side.
Technical details:
- Redash Version: 10.1
- Browser/OS: Linux/Docker/AMI
- How did you install Redash: Docker
If am seeing the following in the DEBUG log:
worker_1 | [2022-03-07 22:18:51,885][PID:53][DEBUG][urllib3.connectionpool] https://webhook.site:443 "POST /526d61ec-64fa-4f58-b38f-f6e176a58381 HTTP/1.1" 200 None
worker_1 | [2022-03-07 22:18:51,891][PID:53][DEBUG][metrics] table=notification_destinations query=select duration=0.79
worker_1 | [2022-03-07 22:18:51,896][PID:53][DEBUG][pypd] Doing HTTP [post] request: https://events.pagerduty.com/v2/enqueue/ - headers: {'Accept': 'application/vnd.pagerduty+json;version=2', 'Authorization': 'Token token=None', 'Content-Type': 'application/json'} - payload: {'routing_key': '[INTEGRATION_KEY]', 'incident_key': '1_242', 'dedup_key': '1_242', 'payload': {'summary': 'Apaleo Custom Payment Alert', 'severity': 'error', 'source': 'redash', 'custom_details': 'Date,Timestamp,DebitedAccountName,ReceiptType,Amount\n2022-03-02,2022-03-02T18:23:55,Liabilities for Reservation JRDXINAN-1,Custom,1.0\n2022-03-02,2022-03-02T18:24:59,Liabilities for Reservation ZRFRQBSA-1,Custom,1.0\n2022-03-02,2022-03-02T18:25:07,Liabilities for Reservation ZRFRQBSA-1,Custom,0.25\n2022-03-02,2022-03-02T18:25:19,Liabilities for Reservation ZRFRQBSA-1,Custom,0.25\n2022-03-02,2022-03-03T00:04:09,Visa Credit Card Payment,Custom,2.0\n2022-03-02,2022-03-03T00:04:28,Credit Card Payment,Custom,1.0\n'}, 'event_action': 'trigger'}
worker_1 | [2022-03-07 22:18:51,898][PID:53][DEBUG][urllib3.connectionpool] Starting new HTTPS connection (1): events.pagerduty.com:443
worker_1 | [2022-03-07 22:18:52,114][PID:53][DEBUG][snowflake.connector.ssl_wrap_socket] OCSP Mode: FAIL_OPEN, OCSP response cache file name: None
worker_1 | [2022-03-07 22:18:52,114][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] ocsp_response_cache_uri: file:///home/redash/.cache/snowflake/ocsp_response_cache.json
worker_1 | [2022-03-07 22:18:52,115][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] OCSP_VALIDATION_CACHE size: 0
worker_1 | [2022-03-07 22:18:52,115][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json
worker_1 | [2022-03-07 22:18:52,115][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] OCSP dynamic cache server RETRY URL: None
worker_1 | [2022-03-07 22:18:52,116][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] Failed to check OCSP response cache file. No worry. It will validate with OCSP server: file: /home/redash/.cache/snowflake/ocsp_response_cache.json, lock directory: /home/redash/.cache/snowflake/ocsp_response_cache.json.lck, error: [Errno 2] No such file or directory: '/home/redash/.cache/snowflake/ocsp_response_cache.json'
worker_1 | [2022-03-07 22:18:52,116][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] Failed to locate OCSP response cache file. No worry. It will validate with OCSP server: /home/redash/.cache/snowflake/ocsp_response_cache.json
worker_1 | [2022-03-07 22:18:52,116][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] validating certificate: events.pagerduty.com
worker_1 | [2022-03-07 22:18:52,117][PID:53][DEBUG][snowflake.connector.ocsp_snowflake] skipping OCSP check: events.pagerduty.com
worker_1 | [2022-03-07 22:18:52,198][PID:53][DEBUG][urllib3.connectionpool] https://events.pagerduty.com:443 "POST /v2/enqueue/ HTTP/1.1" 202 87
worker_1 | [2022-03-07 22:18:52,201][PID:53][WARNING][root] {'status': 'success', 'message': 'Event processed', 'dedup_key': '1_242'}