RDSUnappliedParameters #

Meaning #

Alert is triggered when an RDS instance has unapplied parameter group settings.

Impact #

• RDS instance is running with outdated configuration

  Unexpected changes may be applied after a restart.

Diagnosis #

1. Check instance status

   If the instance is in creating status, the parameter group should be applied automatically by AWS in a few minutes.

2. Identify the RDS parameter group used by the RDS instance

   DB_IDENTIFIER=<db_identifier>
   aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]'
   

   Example

   The db1 instance uses postgres14-primary parameter group. Changes will be applied after a reboot.

   DB_IDENTIFIER=db1
   aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]'
   {
       "DBParameterGroupName": "postgres14-primary",
       "ParameterApplyStatus": "pending-reboot"
   }
   

3. Identify changed parameters

   Search ModifyDBParameterGroup events for the parameter group in AWS Cloudtrail.

   PARAMETER_GROUP_NAME=<RDS parameter group>
   aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyDBParameterGroup | jq --arg PARAMETER_GROUP_NAME "$PARAMETER_GROUP_NAME" '.Events[] | select(.Resources[0].ResourceName == $PARAMETER_GROUP_NAME) | .CloudTrailEvent | fromjson | {userIdentity: .userIdentity, requestParameters: .requestParameters}'
   

   Example

   autovacuum_max_workers parameter on postgres14-primary parameter group was changed to 6

   $ PARAMETER_GROUP_NAME=postgres14-primary
   $ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyDBParameterGroup | jq --arg PARAMETER_GROUP_NAME "$PARAMETER_GROUP_NAME" '.Events[] | select(.Resources[0].ResourceName == $PARAMETER_GROUP_NAME) | .CloudTrailEvent | fromjson | {userIdentity: .userIdentity, requestParameters: .requestParameters}'
   {
       "userIdentity": {
           "type": "AssumedRole",
           "principalId": "AROA5RLBCOJT4ESFJL7UH:terraform",
           "arn": "arn:aws:sts::000123456789:assumed-role/terraform/terraform",
           "accountId": "000123456789",
           "accessKeyId": "ASIA5RLBCOJTTXNLV6UL",
           "sessionContext": {
           "sessionIssuer": {
               "type": "Role",
               "principalId": "AROA5RLBCOJT4ESFJL7UH",
               "arn": "arn:aws:iam::000123456789:role/terraform",
               "accountId": "000123456789",
               "userName": "documentation"
           },
           "webIdFederationData": {},
           "attributes": {
               "creationDate": "2023-09-15T10:48:09Z",
               "mfaAuthenticated": "false"
           }
           }
       },
       "requestParameters": {
           "parameters": [
           {
               "isModifiable": false,
               "applyMethod": "pending-reboot",
               "parameterName": "autovacuum_max_workers",
               "parameterValue": "6"
           }
           ],
           "dBParameterGroupName": "postgres14-primary"
       }
   }
   

Mitigation #

You must restart the RDS instance to fix the pending-reboot apply status.

Important

The following mitigation measures will restart the RDS instance, resulting in a momentary outage. You may consider shutting down the database clients and informing users first.

1. Find a suitable time slot to restart the instance

   Reboot operation can’t be performed if the instance isn’t in the available state. Avoid backup maintenance windows.

2. Apply RDS parameter group changes by restarting the RDS instance

   aws rds reboot-db-instance --no-force-failover --db-instance-identifier ${DB_IDENTIFIER}
   

   This operation is performed asynchronously, it could take several minutes.

   How to see when the restart occurred?

   You can monitor the RDS events

   aws rds describe-events --source-type db-instance --event-categories "availability" --source-identifier ${DB_IDENTIFIER} | jq -r '.Events[] | (.Date + ":" + .Message)'
   

   Example:

   $ aws rds describe-events --source-type db-instance --event-categories "availability" --source-identifier ${DB_IDENTIFIER} | jq -r '.Events[] | (.Date + ":" + .Message)'
   2023-11-29T09:51:13.187000+00:00:DB instance restarted
   

3. Check parameter group apply status is now in-sync.

   aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]'
   

   Example

   $ aws rds describe-db-instances --db-instance-identifier ${DB_IDENTIFIER} --query 'DBInstances[0].DBParameterGroups[0]'
   {
       "DBParameterGroupName": "postgres14-primary",
       "ParameterApplyStatus": "in-sync"
   }
   

Additional resources #

• Working with parameter groups
• Rebooting a DB instance