[10:10:44] jbond42: pcc is complaining trying to compile production catalog of restbase2014 [10:10:47] Error: Failed to compile catalog for node restbase2014.codfw.wmnet: source sequence is illegal/malformed utf-8 [10:11:06] puppet runs fine there, but gives this extra info message: Info: Unable to serialize catalog to json, retrying with pson [10:16:05] vgutierrez: that loosk like a bug can you rais a task and send it my way [10:21:22] done: https://phabricator.wikimedia.org/T238053 [10:29:55] thanks [13:53:27] cdanis: I'm interested in trying to get https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/547596/ merged today — maybe I should schedule it for lunchtime swat? [14:02:42] andrewbogott: he'll be out today FYI [14:03:15] ah, so I see. thanks [14:11:58] godog: [14:12:18] I have a architecture question about prometheus regarding our new Toolforge kubernetes cluster [14:12:55] there are plenty of stuff running inside the cluster, stuff worth scrapping by prometheus [14:13:46] I'm wondering if prometheus servers can be somehow "chained" so I have a server running inside the k8s cluster itself, which them somehow send the metrics to our main tools-prometheus server [14:14:55] we don't allow in the k8s cluster arbitrary access to endpoints, so our main tools-prometheus server may have issues contacting arbitrary /metrics endpoints in the pods running in the cluster [14:15:01] any suggestion? [14:15:25] ttyl, food time [14:21:06] arturo: if allowing tools-prometheus to talk to tools-k8s-prometheus (made up name but you get the idea) then you can configure tools-prometheus to pull all/most metrics via "federation" https://prometheus.io/docs/prometheus/latest/federation/ [14:23:02] I think that's exactly what I was looking for [14:23:11] thanks godog !! [14:24:49] arturo: np! glad it helped, alternatively of course you can add k8s discovery/configuration to tools-prometheus, if you are ok with whitelisting its address. or if there's a proxy in the middle maybe it is possible to filter for address + url path, not sure [16:15:32] godog: I have a patch for you to review [16:19:21] https://gerrit.wikimedia.org/r/c/operations/puppet/+/550506 [16:19:35] also added a few others who are listed as projectadmin in CloudVPS [16:36:03] arturo: kk, I'll take a look today/tomorrow [16:36:10] ack [16:48:15] godog: BTW native service discovery may be a better way to go. RE: prometheus for new k8s cluster [17:07:41] arturo: definitely recommended yeah, the prometheus within k8s I'm assuming would be using native service discovery ? IOW scrape from pods with certain annotations [17:09:57] maybe silly question, but where are the CI docker containers running from? [17:10:05] (which host) [17:14:55] XioNoX: from any of the WMCS VMs acting as jenkins workers, I don't know the hostnames offhand [17:15:17] ok [17:15:45] thx! [18:51:13] seems like https://github.com/respawner/peering-manager can now send peering (and other) emails