[02:30:12] !log anticomposite@tools-bastion-13 tools.stewardbots stewardbots/StewardBot/manage.sh restart # replaying old EventStreams [02:30:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [10:15:30] arturo: dcaro Just a heads-up in case of any unwanted alerts: I'm merging this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1100819 [10:15:59] tappof: thanks 🚢 🇮🇹 [10:15:59] tappof: ack! [10:52:06] tappof: indeed we just had the alert firing [10:52:21] https://usercontent.irccloud-cdn.com/file/SlS4y6la/image.png [10:52:34] and T388379 [10:52:34] T388379: ProbeDown - https://phabricator.wikimedia.org/T388379 [10:52:46] arturo: yes, I've seen [10:54:52] tappof: are you able to investigate? I'm about to jump into a meeting [10:55:17] arturo: yes, I'll check soon [10:55:22] thanks [12:06:28] arturo: We didn't have the IPv6 check before... https://w.wiki/DNG7 AFAICS, Prometheus in eqiad can reach cloudgw2002-dev on the IPv6 VIP, but the reply gets lost somewhere https://snipboard.io/XJ0gn9.jpg [12:06:57] topranks: oh, ok! [12:07:41] we may want to create a ticket to investigate why that happens, and remove the IPv6 check meanwhile [12:10:05] arturo: I think we can use T388379 as the task for this one. I'll link it to the task related to monitoring to keep track of the relationship. [12:10:06] T388379: ProbeDown - https://phabricator.wikimedia.org/T388379 [12:10:29] is it ok for you arturo ? [12:11:26] ok! [12:16:46] arturo: This one is to disable the check in the meantime https://gerrit.wikimedia.org/r/c/operations/puppet/+/1126023 [12:17:19] tappof: +1'd [12:33:33] I'll take a look now... odd I can reach that IP from my home machine but not from prometheus2006 [12:37:30] most likely some filter or similar [12:41:11] yeah it's the cloud-in6 filter on the CRs [12:41:26] they block traffic from the cloud-vrf to "private" WMF ranges [12:41:54] cloud-in4 has an exception to the equivalent rule to allow ICMP [12:42:00] I guess we could mirror it for v6 too [15:24:04] dhinus, caro: I created a tutorial for a nodejs static buildservice. https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_nodejs_tool. Thanks caro for making the static tool guide from which I copied most of the content. 😀 [15:30:02] <3 [15:30:43] dpriskorn: awesome, thank you so much! [15:38:40] Implementing IPv6 is harder than the failed Starship tests :') [17:25:56] Trying to restart a job keeps giving a time out: [17:25:56] requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=30) [17:25:58] Any ongoing issues that might explain this? [17:28:09] !log multichill@tools-bastion-12 tools.geograph Unable to do tools.geograph@tools-bastion-12:~$ toolforge jobs restart geograph-uploader-resumed, keeps giving requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=30) [17:28:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geograph/SAL [17:37:23] dpriskorn: I moved your tutorial and made some small changes to the first section. feel free to tweak it further and thanks again for writing it! [18:00:55] MaartenDammers: nothing I'm aware of, if the issue persists please open a Phab task with details of the job you are trying to restart [18:03:16] dpriskorn: thanks! [18:41:33] !log multichill@tools-bastion-12 tools.geograph Unable to do tools.geograph@tools-bastion-12:~$ toolforge jobs restart geograph-uploader-resumed, keeps giving requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=30) [18:41:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geograph/SAL [19:51:50] !log anticomposite@tools-bastion-13 tools.stewardbots Remove uncommitted changes, deploy b8352a8 (T388292) [19:51:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL