RDSUnappliedParameters #
Meaning #
Alert is triggered when an RDS instance has unapplied parameter group settings.
Impact #
RDS instance is running with outdated configuration
Unexpected changes may be applied after a restart.
Diagnosis #
More
RDS parameter groups have dynamic
and static
parameters:
When you change a dynamic parameter, by default the parameter change takes effect immediately, without requiring a reboot.
When you change a static parameter and save the DB parameter group, the parameter change takes effect after you manually reboot the associated DB instances
When you associate a new DB parameter group with a DB instance, RDS applies the modified static and dynamic parameters only after the DB instance is rebooted
Check instance status
If the instance is in
creating
status, the parameter group should be applied automatically by AWS in a few minutes.Identify the RDS parameter group used by the RDS instance
DB_IDENTIFIER=<db_identifier> aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]'
Example
The
db1
instance usespostgres14-primary
parameter group. Changes will be applied after a reboot.DB_IDENTIFIER=db1 aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]' { "DBParameterGroupName": "postgres14-primary", "ParameterApplyStatus": "pending-reboot" }
Identify changed parameters
Search
ModifyDBParameterGroup
events for the parameter group in AWS Cloudtrail.PARAMETER_GROUP_NAME=<RDS parameter group> aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyDBParameterGroup | jq --arg PARAMETER_GROUP_NAME "$PARAMETER_GROUP_NAME" '.Events[] | select(.Resources[0].ResourceName == $PARAMETER_GROUP_NAME) | .CloudTrailEvent | fromjson | {userIdentity: .userIdentity, requestParameters: .requestParameters}'
Example
autovacuum_max_workers
parameter onpostgres14-primary
parameter group was changed to6
$ PARAMETER_GROUP_NAME=postgres14-primary $ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyDBParameterGroup | jq --arg PARAMETER_GROUP_NAME "$PARAMETER_GROUP_NAME" '.Events[] | select(.Resources[0].ResourceName == $PARAMETER_GROUP_NAME) | .CloudTrailEvent | fromjson | {userIdentity: .userIdentity, requestParameters: .requestParameters}' { "userIdentity": { "type": "AssumedRole", "principalId": "AROA5RLBCOJT4ESFJL7UH:terraform", "arn": "arn:aws:sts::000123456789:assumed-role/terraform/terraform", "accountId": "000123456789", "accessKeyId": "ASIA5RLBCOJTTXNLV6UL", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROA5RLBCOJT4ESFJL7UH", "arn": "arn:aws:iam::000123456789:role/terraform", "accountId": "000123456789", "userName": "documentation" }, "webIdFederationData": {}, "attributes": { "creationDate": "2023-09-15T10:48:09Z", "mfaAuthenticated": "false" } } }, "requestParameters": { "parameters": [ { "isModifiable": false, "applyMethod": "pending-reboot", "parameterName": "autovacuum_max_workers", "parameterValue": "6" } ], "dBParameterGroupName": "postgres14-primary" } }
Mitigation #
You must restart the RDS instance to fix the pending-reboot
apply status.
Important
The following mitigation measures will restart the RDS instance, resulting in a momentary outage. You may consider shutting down the database clients and informing users first.
Find a suitable time slot to restart the instance
Reboot operation can’t be performed if the instance isn’t in the
available
state. Avoid backup maintenance windows.Apply RDS parameter group changes by restarting the RDS instance
aws rds reboot-db-instance --no-force-failover --db-instance-identifier ${DB_IDENTIFIER}
This operation is performed asynchronously, it could take several minutes.
How to see when the restart occurred?
You can monitor the RDS events
aws rds describe-events --source-type db-instance --event-categories "availability" --source-identifier ${DB_IDENTIFIER} | jq -r '.Events[] | (.Date + ":" + .Message)'
Example:
$ aws rds describe-events --source-type db-instance --event-categories "availability" --source-identifier ${DB_IDENTIFIER} | jq -r '.Events[] | (.Date + ":" + .Message)' 2023-11-29T09:51:13.187000+00:00:DB instance restarted
Check parameter group apply status is now
in-sync
.aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]'
Example
$ aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]' { "DBParameterGroupName": "postgres14-primary", "ParameterApplyStatus": "in-sync" }