NTS AWS Managed Services
New Tech Solutions, Inc. (NTS) AWS Managed Services (MS) are made up of a flexible and scalable team to meet customers’ needs for certified AWS Solutions Architects, SecOps, DevOps, and system administrators. NTS’ MS team is thoughtfully designed to augment in-house customer IT staff and aligning with business and technical needs. NTS’ expertise and tool sets can deliver consistent monitoring, alerting, optimizing, securing, cost saving, and deploying of AWS workloads for any sized environment. An example of this can be seen through a customer requirement to regularly patch a mission critical application part of an AWS Auto Scaling Group (ASG) with zero down-time SLA for the maintenance. NTS’ MS team
Customers challenge
The Challenge
The customer was relying on a lengthy and error prone task to manually apply monthly OS security patches. Not only was this process difficult but it left the application limited as the physical servers were taken out of rotation to complete the patching. Now that the customer’s application was migrated to AWS they turned to NTS MS to optimize this process while maintaining security compliance and enhancing application availability during maintenance. With the application workload in the cloud the customer was still relying on similar manual steps to patch an AMI and update the instances in the ASG Launch Configuration (LC) with it. Manual steps which were not tied to a set schedule and included: manually launching an instance from a Golden AMI copy, patch the instance, stop the instance, create an AMI copy of the instance, manage AMI copies and instances, find ASG and manually copy new AMI ID, and finally OS reboots one at a time to try and keep application availability at peak performance.
NTS SOLUTION
The Solution
Understanding the security and availability requirements of the application in AWS, NTS’ MS team elected to use AWS Lambda, Automation Document, SSM Patch Manager, and custom IAM Roles to patch the Red Hat Enterprise Linux (RHEL) instances in the ASG. To successfully automate this process using AWS services, NTS’ MS team created the following end-to-end solution:
AWS IAM Roles to:
- Encrypt/decrypt the instances (ensuring data at rest is preserved)
- Allow SSM to run Automation Document (Zero-Trust authentication)
- Lambda function to update ASG LC (Zero-Trust resource interaction)
- SNS Topic Pass Role tied to SSM Maintenance Window & Automation (authenticated notifications)
AWS Lambda
- Create Lambda function with Python runtime to create new LC using the updated AMI from Automation Document (create new LC using newly patched AMI, apply LC to target ASG)
AWS Systems Manager:
- Create custom JSON Automation Document to Patch the RHEL Golden AMI and update the ASG LC with new AMI ID (orchestrates which resource IDs, Instance Type, subnets, Security Group, IAM Instance Profile to use, and Lambda function to update the ASG all in a particular order)
- Maintenance Window with Cron expression to run at a scheduled time each month (dictates when Automation Document is executed)
- Use an approved RHEL Patch Baseline for Critical, Important, and Moderate Security patches (apply approved RHEL patches per the customer requirements)
SNS Topic
- Register notifications to send upon Success or Failures of Automation to MS team
Once the Automation steps are executed successfully per the Maintenance window an SNS Topic notifies the NTS MS team to initiate an ASG Refresh. The Refresh is a controlled manner in which instances in the ASG are replaced (deregistered) with the newly patched AMIs from the automation process. New instances are initialized and introduced to the ASG one at a time to conform with the requirement that a minimum of 3 application servers are available at all times.
NTS Results
The Results
NTS’ automation of AWS and Vendor patching has solidified the customers security posture related to OS patching. This once labor-intensive manual process is now fully automated saving the customer time and money while allowing the critical application to remain highly available and scalable during maintenance. The resulting success the NTS MS team was able to provide gave the customer multiple technical and business victories.
- ü 0 missed patching cycles
- ü 80% improvement to application availability during AWS maintenance
- ü Latest protection from Vendor OS Security patch releases
- ü Over 100 hours of Cloud Systems Administrator work saved annually
- ü Reduced attack surface from bad actors
