vmalert

Rule: TooManyRemoteWriteErrorsok

Expr

sum(rate(vmagent_remotewrite_retries_count_total[5m])) by(job, instance, url) > 0

For

900 seconds

Labels

severity=warning

Annotations

dashboard:

http://localhost:3000/d/G7Z9GzMGz?viewPanel=61&var-instance={{ $labels.instance }}

description:

Vmagent fails to push data via remote write protocol to destination "{{ $labels.url }}" Ensure that destination is up and reachable.

summary:

Job "{{ $labels.job }}" on instance {{ $labels.instance }} fails to push to remote storage

Group

11604620448585840090

Last 20 updates:

Updated at	Duration	Executed at	cURL
2026-04-29T16:29:18Z	0.001s	2026-04-29T16:29:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777480140'
2026-04-29T16:28:48Z	0.001s	2026-04-29T16:28:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777480110'
2026-04-29T16:28:18Z	0.001s	2026-04-29T16:28:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777480080'
2026-04-29T16:27:48Z	0.001s	2026-04-29T16:27:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777480050'
2026-04-29T16:27:18Z	0.001s	2026-04-29T16:27:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777480020'
2026-04-29T16:26:48Z	0.001s	2026-04-29T16:26:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479990'
2026-04-29T16:26:18Z	0.001s	2026-04-29T16:26:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479960'
2026-04-29T16:25:48Z	0.001s	2026-04-29T16:25:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479930'
2026-04-29T16:25:18Z	0.001s	2026-04-29T16:25:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479900'
2026-04-29T16:24:48Z	0.000s	2026-04-29T16:24:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479870'
2026-04-29T16:24:18Z	0.001s	2026-04-29T16:24:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479840'
2026-04-29T16:23:48Z	0.001s	2026-04-29T16:23:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479810'
2026-04-29T16:23:18Z	0.001s	2026-04-29T16:23:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479780'
2026-04-29T16:22:48Z	0.001s	2026-04-29T16:22:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479750'
2026-04-29T16:22:18Z	0.001s	2026-04-29T16:22:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479720'
2026-04-29T16:21:48Z	0.001s	2026-04-29T16:21:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479690'
2026-04-29T16:21:18Z	0.001s	2026-04-29T16:21:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479660'
2026-04-29T16:20:48Z	0.001s	2026-04-29T16:20:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479630'
2026-04-29T16:20:18Z	0.001s	2026-04-29T16:20:18Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479600'
2026-04-29T16:19:48Z	0.001s	2026-04-29T16:19:48Z	curl -X POST -H 'Content-Type: application/json' 'http://victoriametrics:8428/api/v1/query?query=sum%28rate%28vmagent_remotewrite_retries_count_total%5B5m%5D%29%29+by%28job%2C+instance%2C+url%29+%3E+0&step=300s&time=1777479570'