Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Prometheus: how does this query work exactly?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Prometheus: how does this query work exactly?

I have implemented a trigger for KEDA event based autoscaling in Kubernetes for our backend, which is a Rails API. This trigger uses a custom Prometheus metric which is basically the queue time, so the goal is to scale replicas based on how long requests wait in the queue before being performed.

Now, I got some help recently from someone on a Slack community to create a query to use this metric in a trigger, and this person suggested this:

scalar(kube_horizontalpodautoscaler_status_current_replicas{horizontalpodautoscaler="keda-hpa-{{ .Values.service.name }}-web"}) * histogram_quantile(0.95, sum by (le)(rate(ruby_queue_latency_bucket{service="{{ .Values.service.name }}-web"}[1m]))) OR on() vector(0)

I am trying to understand how this works exactly but that person is no longer replying, maybe he's busy or he's left the community, dunno. I am familiar with the basics of PromQL so I am a bit confused by this.

kube_horizontalpodautoscaler_status_current_replicas is the number of replicas currently available and fired up by the horizontal pod autoscaler (HPA), and this HPA is managed by the KEDA ScaledObject.

histogram_quantile(..) is the p95 of the metric I was mentioning, the queue time.

Why did he suggest I multiply the replicas with the p95 of the queue time? The HPA will scale based on the average of the values returned for that query for each replica.

Also what does OR on() vector(0) do?

Another thing: he suggested that the threshold for the trigger be 25, which in this case means 25 milliseconds wait.

I would appreciate if anyone with more experience with Prometheus could clarify this stuff. Thanks!

Comments

  • HxxxHxxx Member
    edited September 2022

    There are better places to ask complex questions like these. Most people here are present just for the deals and don't have that kind of knowledge you are looking for.

    Check stackexchange, maybe devops?

    Most likely it will be derailed with some non-sense.

  • @vitobotta said: Also what does OR on() vector(0) do?

    This part treats null as zero.

  • @Hxxx said:
    There are better places to ask complex questions like these. Most people here are present just for the deals and don't have that kind of knowledge you are looking for.

    Check stackexchange, maybe devops?

    Most likely it will be derailed with some non-sense.

    I guess there are people here who use Prometheus etc on their servers??

    @itsnotv said:

    @vitobotta said: Also what does OR on() vector(0) do?

    This part treats null as zero.

    Thanks! It looked obscure to me but it makes sense. Thanks

Sign In or Register to comment.