Skip to main content

LFX Mentorship: Integration Tests for OpenCost Enterprise Readiness Part (2 / 2)

Updated 5 min readMarkdown
LFX Mentorship: Integration Tests for OpenCost Enterprise Readiness Part (2 / 2)

The General Idea behind Cost

There are three important variables in the equation. One is the running time of that resource, and the second is the resource in terms of a measurable unit value. I faced issues tackling the first one; more about it later.

  • The resource is the maximum of the requested value and the used value at any given instance.

  • Run Time is the total running of for that resource, including idle times in some cases.

  • Resource Type: either a Container, Pod, or Namespace. Costs are generally calculated on a Container level and aggregated to a Pod or a namespace level.

Here is a general outlook on cost calculation for RAM Usage (I like this one because this test flakes a lot even today)

for _, promResponseItem := range promResponse.Data.Result {
                if promResponseItem.Metric.Container == "" {
                    continue
                }
                // Get containerRunTime by getting the pod's (parent object) runtime.
                pod, ok := podMap[promResponseItem.Metric.Pod]
                if !ok {
                    continue
                }

                containerRunTime := pod.Window.RunTime()

                ramUsageAvgPod, ok := ramUsageAvgNamespaceMap[promResponseItem.Metric.Namespace]
                if !ok {
                    ramUsageAvgNamespaceMap[promResponseItem.Metric.Namespace] = &RAMUsageAvgAggregate{
                        PrometheusUsageAvg: promResponseItem.Value.Value * containerRunTime,
                        AllocationUsageAvg: 0.0,
                        Window: &api.Window{
                            Start: pod.Window.Start,
                            End:   pod.Window.End,
                        },
                    }
                    continue
                }
                ramUsageAvgPod.PrometheusUsageAvg += promResponseItem.Value.Value * containerRunTime
                ramUsageAvgPod.Window = api.ExpandTimeRange(ramUsageAvgPod.Window, pod.Window)
            }

            // windowRunTime := queryEnd.Sub(queryStart).Minutes()
            for _, ramUsageAvgProm := range ramUsageAvgNamespaceMap {
                ramUsageAvgProm.PrometheusUsageAvg = ramUsageAvgProm.PrometheusUsageAvg / ramUsageAvgProm.Window.RunTime()
            }

Running Times is not simple

While you make decisions around cost, you might be tempted to visualize a dashboard with only long-running namespaces. It is definitely a valid assumption to start with.

I started to feel the grunt of my ignorance later for the following reasons:

  1. How do you handle Pod UIDs?

  2. How do you handle short-lived namespaces?

POD UIDs are tricky; while OpenCost does not support PodUIDs, it does provision separate logical calls for calculating UID costs. Pod UIDs are generally of two types: either the Pod is a replica, or the Pod crashed and it restarted.

We introduced a chaos monkey later into the mentorship because my RAM tests started to fail. Turns out shorter-lived namespaces, which were created whenever a new GitHub Action was running, cannot be treated in the same way as long-running ones. To understand and protect our tests, we decided to kill pods randomly to reproduce short-lived pods.

Interesting Challenges I Crossed

Prometheus Queries run in a particular interval period. If the run time of the pod is proportional to the windowSize, cost calculations can end up flaky

Prom 3.0 Changes

OpenCost does a couple of extra manipulations to account for the n+1 response values.

func (pds *PrometheusMetricsQuerier) durationStringFor(start, end time.Time, minsPerResolution int) string {
    dur := end.Sub(start)

    // If using a version of Prometheus where the resolution needs duration offset,
    // we need to apply that here.
    //
    // E.g. avg(node_total_hourly_cost{}) by (node, provider_id)[60m:5m] with
    // time=01:00:00 will return, for a node running the entire time, 12
    // timestamps where the first is 00:05:00 and the last is 01:00:00.
    // However, OpenCost expects for there to be 13 timestamps where the first
    // begins at 00:00:00. To achieve this, we must modify our query to
    // avg(node_total_hourly_cost{}) by (node, provider_id)[65m:5m]
    if pds.promConfig.IsOffsetResolution {
        // increase the query time by the resolution
        dur = dur + (time.Duration(minsPerResolution) * time.Minute)
    }

    return timeutil.DurationString(dur)
}
// DurationString converts a duration to a Prometheus-compatible string in
// terms of days, hours, minutes, or seconds.
func DurationString(duration time.Duration) string {
    const (
        SecsPerMin    = 60.0
        SecsPerHour   = 3600.0
        SecsPerDay    = 86400.0
        MinsPerHour   = 60.0
        MinsPerDay    = 1440.0
        HoursPerDay   = 24.0
        HoursPerMonth = 730.0
        DaysPerMonth  = 30.42
        Day           = time.Hour * 24.0
        Week          = Day * 7.0
    )

    durSecs := int64(duration.Seconds())

    durStr := ""
    if durSecs > 0 {
        if durSecs%SecsPerDay == 0 {
            // convert to days
            durStr = fmt.Sprintf("%dd", durSecs/SecsPerDay)
        } else if durSecs%SecsPerHour == 0 {
            // convert to hours
            durStr = fmt.Sprintf("%dh", durSecs/SecsPerHour)
        } else if durSecs%SecsPerMin == 0 {
            // convert to mins
            durStr = fmt.Sprintf("%dm", durSecs/SecsPerMin)
        } else if durSecs > 0 {
            // default to secs, as long as duration is positive
            durStr = fmt.Sprintf("%ds", durSecs)
        }
    }

    return durStr
}
// Adjust Window Size for a 24h window
unit := time.Hour
duration := 24 * unit
now := time.Now().UTC()
adjusted := now.Add(1 * unit)
end := adjusted.Truncate(unit)
start := end.Add(-duration)

This was one of the hardest ideas I struggled with throughout the mentorship. I needed to adjust the window account for the extra value because a 24-hour window to the opencost API is not the same 24h window to Prometheus.

Since the pod runtime was quiet small, differences that woudn’t affect long running pods can affect my calculations here.

Unmarshalling JSoN

I’m proud of this one. It was hard to design structs for non-JSON confirming API responses. To tackle this, you need to implement a custom unmarshalling function that runs on part of the response, translates to a byte object, and maps the desired values to your structs.

Debugging

I was suggested to set up an OpenCost Dev environment to cross-verify the log messages that read the OpenCost windows and Prometheus windows. I was able to figure things out with old-fashioned print statements. But here are two general themes I picked up while spending time stuck on the problem

  • Ensure code runs - try to get the code to run without throwing syntax issues.

  • Fix the breaking objects one by one, which will help you make observations steadily. You don’t have to fix everything immediately; you can put one object under the microscope and fix that. Turns out, most of the time, it is a domino effect, and other tests will also start to pass.

What’s Next

I’ll be posting my updates on the GitHub Plugin.

No comments yet. Be the first to comment.