manheim_c7n_tools.errorscan module

class manheim_c7n_tools.errorscan.CustodianErrorReporter(config, region_name)

Bases: object

Scan and report on CW Metrics/Logs errors for c7n lambdas

ALL_ERROR_FUNCTIONS = re.compile('^cloud-custodian.*')

ALL_ERROR_LOG_RE = re.compile('.*(ERROR|WARNING).*')

INTERVAL = 86400

How far to look back in logs and metrics, in seconds

INTER_FUNC_SLEEP = 3.0

Amount of time (float seconds) to sleep between checking each function, to try to avoid API rate limiting.

INVL_DESC = 'day'

Human-readable description of the interval

METRIC_PERIOD = 86400

What period to request CloudWatch metrics for

_ack_sqs()

Delete (ack) all SQS messages in self._sqs_rcpts.

_check_function(func_name, never_match_re=None)

Check health of one Lambda function. Print information on it to STDOUT. Return True for healthy, False if errors/failures.

Parameters

• func_name (str) – Lambda function name to check

• never_match_re (re) – Regex for logs to NEVER return, even if they match always_match_re.

Returns

whether the function had errors/failures

Return type

bool

_get_sqs_dlq()

Pull all messages from the SQS Dead Letter Queue. Add the failed Lambda RequestIDs to self._failed_request_ids and the SQS Reciept Handles to self._sqs_rcpts.

_sqs_arn_to_url(arn)

Find the URL for an SQS Queue given its ARN.

Parameters

arn (str) – the ARN of the queue

Returns

the URL of the queue

Return type

str

run(never_match_re=None)

collect and report on all cloud-custodian Lambda errors

class manheim_c7n_tools.errorscan.LambdaHealthChecker(func_name, region_name, logs=None, cw=None)

Bases: object

Class for checking Lambda func health via CloudWatch

NO_OWNER_EMAIL_LOOKUP_WARNING = re.compile('.*(ERROR|WARNING).*unable to lookup owner email.*Please configure LDAP or org_domain')

_get_cw_log_stream(group_name, stream_name, start_ts, end_ts)

Return all log messages from the specified stream at or after ts.

Parameters

• group_name (str) – CloudWatch log group name

• stream_name (str) – CloudWatch log stream name

• start_ts (int) – timestamp in milliseconds to return logs after

• end_ts (int) – timestamp in milliseconds to return logs before

Returns

Return type

list

static find_matching_func_names(filter, region_name, client=None)

Return a list of all Lambda functions with names that either start with filter (if filter is a string) or match filter (if filter is a re.RegexObject).

Parameters

• filter (str re.RegexObject) – lambda function name filter

• region_name (str) – region name to run against

• client (boto3.client) – boto3 Lambda client, or None to create new

Returns

list of matching Lambda function names

Return type

list

get_cloudwatch_logs(interval=86400, group_name=None)

Get CloudWatch logs for the last interval seconds. The log group name defaults to /aws/lambda/{func_name} if left at the default of None.

Parameters

• group_name (str) – CloudWatch logs group name. If left at default of None, defaults to /aws/lambda/{func_name}.

• interval (int) – how far back in logs to look, in seconds

Returns

list of log entry dicts, sorted by timestamp

Return type

list

get_cloudwatch_metric_sums(interval=86400, period=86400)

Return a dict of CloudWatch Metrics for this Lambda function, summed over interval. Keys are metric names (“Errors”, “Throttles”, “Invocations”) and values are sums of each period-period datapoint, for the past interval seconds.

For further information on these metrics, see: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ lam-metricscollected.html

Parameters

• interval (int) – how many seconds of historical data to request

• period (int) – the metric collection period to request from CloudWatch

Returns

dict of metric name to sum for the last interval seconds

Return type

dict

get_filtered_logs(request_ids, interval=86400, group_name=None, always_match_re=None, never_match_re=None)

Get CloudWatch logs for the last interval seconds and return only those entries with messages matching filter_re.

Parameters

• request_ids (list) – list of str request IDs to get logs for

• group_name (str) – CloudWatch logs group name. If left at default of None, defaults to /aws/lambda/{func_name}.

• interval (int) – how far back in logs to look, in seconds

• always_match_re (re) – Regex for logs to ALWAYS return

• never_match_re (re) – Regex for logs to NEVER return, even if they match always_match_re.

Returns

dict of request_id to list of log entry dicts

Return type

dict

req_id_re = re.compile('^(START|END|REPORT|\\S+\\s\\S+)\\s([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}).*')

class manheim_c7n_tools.errorscan.UTC

Bases: datetime.tzinfo

dst(dt)

datetime -> DST offset as timedelta positive east of UTC.

tzname(dt)

datetime -> string name of time zone.

utcoffset(dt)

datetime -> timedelta showing offset from UTC, negative values indicating West of UTC

manheim_c7n_tools.errorscan._name_value_dict(lst)

Given a list (lst) containing dicts with Name and Value keys, return a single dict of Name -> Value.

manheim_c7n_tools.errorscan.main()

manheim_c7n_tools.errorscan.parse_args(argv)