AWS Cloud Financial Management for Akkodis Public Sector Managed Services Client

The client is a public sector organisation in Australia, who has engaged Akkodis for complete Enterprise Cloud Managed Services.

Money falling from the clouds

Cloud spend tracking and optimisation can be complex in large organisations, with many workloads all executing across multiple environments, with continuous innovation and improvement activities being undertaken.

The client holds Akkodis responsible for Analysing, interpreting, recommending, and executing cost optimisation activities to ensure that public money is not spent unnecessarily.

Akkodis provides three-way optimisation of total operational cost:

  1. Architectural Review and alignment
  2. Per-workload (per service team) micro-optimisations as part of continual operations
  3. Top-down cost optimisation reviews and billing execution

Akkodis facilitates coordination across multiple service teams, each of which have responsibility for different (sets of) applications, by way of an Architectural Review Board before workloads go live or have major changes to their architecture. These governance meetings try to validate each service team’s technical architecture and application technology stack to find synergies between them that help reduce unnecessary technical solutions that will further stretch the team and/or increase cost.

In many cases, Akkodis found that a common database option for workloads was Amazon RDS using the Postgres engine. Rather than dilute the Service Team’s operational knowledge of various databases, RDS Postgres was selected as a preferred technical component.

Akkodis also leveraged the Well-Architected Principles of cloud engineering to review these workloads.

One part of the review is to determine in which AWS Account(s) the workload will run. While common shared infrastructure accounts exist, smaller workloads are often grouped together, where absolute cost transparency is not required. For any larger workload, which must retain itemised cloud costs, and/or be segregated from other workloads to avoid “noisy neighbour” scenarios where adjacent workloads may blow through service quotas, or where there is a potential for workloads to be spun off to other organisations, separate sets of AWS Accounts are used.

Akkodis always separated non-production environments from production; typically for bespoke software development this is four or more environments: Development, Test, User Acceptance testing (UAT), and Production. With this level of separation, costs can be determined for just production environments, non-production, or by all environments for a particular workload.

Workloads are deployed by CloudFormation templates, with parameters to create unique differences between environments, such as instance sizes or block storage partition capacities, to further optimise the costs.

These templates also often contain various alarms around standard CloudWatch metrics for the resources being used, as well as custom CloudWatch metrics that indicate service health for each workload. Limits and thresholds for the metrics are either uniform for all environments for the workload, keyed from the environment name, or parameterised directly from the CloudFormation template.

The review also includes anticipating the implied cost for the workload, confirming the constraints and technology choices adopted. At all points, the adoption of managed platform services is positioned to minimise the manual effort overhead that needs to be applied to implement, operate & maintain the full tech stack.

As part of a durability and DevOps delivery, the majority of EC2 instances were deployed in AutoScale groups.

After adoption of AutoScale, for non-production environments those AutoScale groups were resized to zero instances outside of business hours. Not only did this provide a massive cost optimisation for our client but ensured that total service recoverability was automated by resizing the AutoScale group.

Akkodis also reviews the applicability of Serverless architectures in client solutions to further accelerate the value realisation.

Each service team, providing regular maintenance and management of applications, is tasked with reviewing resource utilisation, and the continual improvement as AWS Cloud services evolve. These continual Cost-focused innovations include (but are not limited to):

  • Modernisation of EC2, RDS and other services’ instance type families, particularly as newer more performant instance types are introduced with a better price point. However, this is done in union with top-down billing activities.
  • Optimisation of EC2, RDS and other services’ instance type sizes, finding a balance between CPU in use and memory in use
  • Adoption of more cost-efficient CPU architectures, such as ARM (Graviton)
  • Optimisation of black storage: archive & deletion of data from block storage to cheaper storage tiers (such as S3)
  • Removal of unused block storage volumes, preserved via snapshots if data retention is required
  • Termination of unused EC2 instances
  • Modernisation of block storage types, as these evolve over time, and the price point adjusts (for example, the move from GP2 to GP3 storage).
  • The move to use managed services over self-managed, to reduce the amount of undifferentiated heavy lifting by our most expensive resources: people.

A simple saving is to quiesce any non-non-production workload outside of core business hours. With typical working hours being around 40 – 60 hours per week (168 hours) across all team members, the general saving using on demand pricing is between 64% - 76%. Furthermore, this blackout can force certain best practices, such as disposable infrastructure via AutoScale – a benefit not just for cost, but scaling, disaster recovery, and helping accelerate future maintenance operations like re-baselining EC2 instances to new operating system versions.

Our top-down review starts with the Consolidated billing totals. All client workloads consolidate their billing to the same AWS Account (as part of orchestration by Control Tower).

On a month-by-month basis, and an AWS account-by-account approach, costs are reviewed and compared with previous months. After any known activities or events that could incur billing differences are removed from the monthly total, these are compared to previous months. Variations that are significant are then raised for deeper investigation by individual service teams to confirm these changes are expected and in line with changing usage patterns.

Akkodis then reviews the various compute and storage options at a top-down approach, understanding the amount that is covered by Reservations and Savings Plans.

With traditional Reservations, as coverage expires, this is lined up with larger architectural changes and other modifications, due to the inflexibility of the Reservation system.

With Savings Plans providing more flexibility in changing compute types, the tendency has been to take longer 3-year commitments under this option. With our client’s workloads we anticipate savings averaging 48% over a using a no-upfront option, which provides similar savings versus partial and all-upfront options, but without the inconvenience of administrative overhead in preparing for and validating high short-term spend.

However, at this time, Savings Plans do not cover a large aera of our client’s usage, being Relational Database Service (RDS). Given the inflexibility here, we continue to preference 1 year periods, with either partial or no-upfront costs.

Our experience in Cost Allocation Tagging over a decade has shown that various parts of the business changes over time: project names that described a service become irrelevant, and some projects have a short life span. Even Cost Centres allocated by finance teams change over time.

Our current approach to move towards tagging using a standardised Application Name (across all environments and AWS accounts) instead of Project names or Cost Centres. This lets tags change less frequently while roll-ups for reporting happen dynamically on the output of AWS Cost Allocation Tagging reports.

As we cautiously move towards this approach, Tagging Policies as part of AWS Organisations become important.

Our client has seen superb value realisation of the AWS Cloud, guided by the senior Akkodis AWS Cloud team. By being proactive and strategic on cost savings, we ensure that the promise of cost-effective cloud is realised. Our client is highly satisfied with our service, both from the individual workload architecture, implementation, and operations, but also the top-down cloud governance and FinOps management to ensure ongoing effectiveness and dexterity as conditions change.