[00:17:46] Is there an easy way to get the central auth information for multiple users at once using the API and/or the database? [00:34:03] (03CR) 10Jean-Frédéric: [C: 032] Make two cosmetic fixes [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293697 (owner: 10Lokal Profil) [00:34:50] (03Merged) 10jenkins-bot: Make two cosmetic fixes [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293697 (owner: 10Lokal Profil) [00:58:27] Magog_the_Ogre, the database yes, definitley [00:58:29] definitely* [00:59:37] Krenair, I was asking for how, not simply if it exists. [00:59:54] Maybe a document on where to start with the database? [01:00:10] yeah hang on, I'm still looking at the API bit for you [01:00:25] https://meta.wikimedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=globalallusers&agufrom=Magog [01:00:29] OK just making sure :) [01:00:32] hit 'make request' [01:01:04] you probably want to add stuff like aguprop from list=globalallusers [01:01:30] Create the page "Aguprop" on this wiki! [01:01:30] There were no results matching the query. [01:01:42] what are you doing [01:02:23] go to my url, click 'list=globalallusers' [01:02:28] scroll to aguprop [01:02:41] https://commons.wikimedia.org/wiki/User_talk:Magog_the_Ogre#The_new_user_gallery [01:03:27] ah there I see it [01:03:32] you want edit count information? [01:03:58] Yes I want the global edit count information for a very specific set of users. [01:04:31] don't think that's provided [01:06:29] I see something about a generator [01:07:42] If you don't know I can send an email to the list [01:31:24] 10Labs-Other-Projects, 06Discovery, 06Maps: Receiving warning emails about failed puppet runs - multiple instances - https://phabricator.wikimedia.org/T135942#2375183 (10Andrew) 05Open>03Resolved That's great! As always, this is due to various terrible NFS problems which are difficult to deal with compr... [08:14:45] andrewbogott: tools-bastion-03 is down again [08:15:12] is here anybody bringing it up again? [08:25:13] 06Labs, 10Tool-Labs: several stucks of tools-bastion-03 - https://phabricator.wikimedia.org/T137683#2375382 (10doctaxon) [08:41:02] 06Labs, 10Tool-Labs: several stucks of tools-bastion-03 - https://phabricator.wikimedia.org/T137683#2375382 (10valhallasw) Connection hangs at ``` debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_... [09:04:37] hmm trying to create a new instance for Phragile on https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance but don't see the link although I'm admin o.O [09:05:08] and going to https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=create&project=phragile®ion=eqiad directly does not let me choose "Instance type" or "image type" [09:05:45] ( WMDE-Fisch ) [09:14:56] 06Labs, 10Tool-Labs: several stucks of tools-bastion-03 - https://phabricator.wikimedia.org/T137683#2375485 (10valhallasw) Server load is consistent with an NFS failure: {F4160258} I have rebooted the host. [09:19:05] 06Labs, 10Tool-Labs: several stucks of tools-bastion-03 - https://phabricator.wikimedia.org/T137683#2375488 (10valhallasw) 05Open>03Resolved a:03valhallasw The host is back on-line. [10:09:57] 06Labs, 10Labs-Infrastructure, 10DBA: Add shared edit rights between eranbot and copypatrol tools - https://phabricator.wikimedia.org/T137691#2375635 (10Niharika) [10:25:31] (03CR) 10Jean-Frédéric: Import messages from https://github.com/Krinkle/intuition (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [10:29:17] (03CR) 10Jean-Frédéric: [C: 032] Add testing infrastructure with npm/grunt [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (owner: 10Jean-Frédéric) [10:30:19] (03Merged) 10jenkins-bot: Add testing infrastructure with npm/grunt [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (owner: 10Jean-Frédéric) [10:42:39] (03CR) 10Lokal Profil: Import messages from https://github.com/Krinkle/intuition (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [10:43:04] yuvipanda: I can't create new instances for phragile even though I am admin with WMDE-Fisch - any idea why? for my colleagues its working :-/ [10:58:48] (03CR) 10Jean-Frédéric: "Answered one comment inline." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [11:04:57] CFisch_home: have you tried logging out and in again? Wikitech gets confused sometimes. [11:05:50] valhallasw`cloud: n1 that did it [11:05:52] thanks [11:07:31] valhallasw`cloud, do you know if there is an easy way to go from internal ip to tool/user/vm? [11:07:52] (sorry, not very familiar with the whole setup) [11:08:14] jynus: you can get the VM hostname in most cases with nslookup [11:09:11] jynus: but mapping activity to a specific user is less trivial. I think identd should work if you know the port number [11:10:43] difficult, we found some high production load potentially coming from labs [11:11:26] and the clues we have are very vague, probably due to our own proxy usage, etc. [11:11:48] jynus: this is api.php load? [11:11:55] no [12:12:37] (03PS6) 10Lokal Profil: Import messages from https://github.com/Krinkle/intuition [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) [12:16:36] jynus did you find what you were looking for? [12:18:52] (03PS7) 10Lokal Profil: Import messages from https://github.com/Krinkle/intuition [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) [12:30:44] (03CR) 10Jean-Frédéric: [C: 032] Import messages from https://github.com/Krinkle/intuition [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [12:35:40] (03Merged) 10jenkins-bot: Import messages from https://github.com/Krinkle/intuition [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [13:05:53] (03CR) 10Hashar: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (owner: 10Jean-Frédéric) [13:06:25] (03CR) 10Hashar: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [13:06:27] (03CR) 10Hashar: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [13:06:30] (03CR) 10Hashar: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293889 (https://phabricator.wikimedia.org/T134764) (owner: 10Jean-Frédéric) [13:06:56] (03CR) 10jenkins-bot: [V: 04-1] Add CSS linting via stylelint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [13:18:17] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [13:20:28] 06Labs, 10Tool-Labs, 10DBA, 06Operations, 10Traffic: Antigng-bot improper non-api http requests - https://phabricator.wikimedia.org/T137707#2376097 (10jcrespo) [13:21:57] (03PS3) 10Jean-Frédéric: Add CSS linting via stylelint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 [13:26:01] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2376121 (10hashar) [13:26:46] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2368900 (10hashar) Occurred again https://integration.wikimedia.org/c... [13:26:50] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2368900 (10MoritzMuehlenhoff) Is there a way to re-trigger the CI job... [13:32:50] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2376129 (10hashar) >>! In T137460#2376125, @MoritzMuehlenhoff wrote:... [13:35:40] !log deployment-prep Fixed puppet on -tin by symlinking eventbus key to eventlogging in -puppetmaster:/var/lib/git/labs/private/modules/secret/secrets/keyholder [13:35:40] Please !log in #wikimedia-releng for beta cluster SAL [13:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [13:53:12] doctaxon: it looks ok to me now — I got a 'down' alert and then an 'up' alert five minutes later. I don't know why. [13:53:42] !log deployment-prep kicked deployment-salt via nova for Krenair [13:53:42] Please !log in #wikimedia-releng for beta cluster SAL [13:53:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [13:54:24] andrewbogott: that's the reason: https://phabricator.wikimedia.org/T137683 [13:55:08] hm, yep [13:55:14] I don't think that's what killed it the other times :( [13:55:38] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2376153 (10BBlack) The 10% random-failure patch went live ~13:00 UTC. Merlbot is temporarily excluded from the 10% (with a match on Merlbot's UA... [13:55:52] so we've not got the reason why [13:56:36] andrewbogott: is it possible to monitor tools-bastion-03 somehow? [13:57:00] doctaxon: it is monitored. It just doesn't get me out of bed if it dies :) [13:57:19] ya, ;) [14:55:35] 06Labs, 10Tool-Labs, 10DBA, 06Operations, 10Traffic: Antigng-bot improper non-api http requests - https://phabricator.wikimedia.org/T137707#2376258 (10Antigng_) My bot was using /w/index.php?action=raw to fetch the content of each page/redirect at zhwiki, then it will do some simple search/replace/templa... [15:17:56] 06Labs, 10Tool-Labs, 10DBA, 06Operations, 10Traffic: Antigng-bot improper non-api http requests - https://phabricator.wikimedia.org/T137707#2376361 (10Joe) @Antigng_ you might not have seen anything go wrong, but your bot was accounting for 50% of the uncached requests to our backends or more. It's a cle... [15:36:53] 06Labs, 10Labs-Infrastructure: Rebalance labvirt1010 - https://phabricator.wikimedia.org/T137719#2376455 (10Andrew) [15:41:53] 06Labs: confirm that new base labs base image is adequate for kubernetes &c. - https://phabricator.wikimedia.org/T134944#2376494 (10yuvipanda) I'm creating tools-worker-1013 just now and provisioning it with k8s and checking. [15:55:02] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2376525 (10Bmueller) @BBlack, thanks for the info & action, that gives everyone a bit more time. About a stronger FOSS policy on tools/bots - the... [16:06:40] !log deployment-prep Rebooted deployment-ircd, it was stuck somehow [16:06:40] Please !log in #wikimedia-releng for beta cluster SAL [16:06:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [16:07:12] 10Labs-project-extdist, 10VisualEditor: extdist tarball generator is erroring on VisualEditor REL1_23 - https://phabricator.wikimedia.org/T121748#2376590 (10Jdforrester-WMF) Tagging with VE so we can keep track of this. [16:20:11] 06Labs: confirm that new base labs base image is adequate for kubernetes &c. - https://phabricator.wikimedia.org/T134944#2376649 (10yuvipanda) Running into lots of ``` - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [memory]] ``` which makes this not work. [16:24:42] yuvipanda: Is ^ the same bug that 4.2 had? [16:25:29] andrewbogott no, but an earlier one that you fixed, and I know joe touched it recently for mw appservers + jessie [16:25:40] hm [16:43:36] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2376676 (10faidon) @brion, any news? [16:45:26] The image you're running was built before _joe_'s changes, so it can't be related to that. [16:45:35] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2376681 (10brion) Seems ok lately, haven't noticed any problems last week. [16:46:45] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2343755 (10yuvipanda) Through the proxy or the public IP? [17:01:49] 06Labs, 06Operations, 10netops: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2376702 (10brion) Either. You can have the IP back, I guess, doesn't seem to make any difference. [17:03:42] yuvipanda: i've disassociated the ip [17:05:58] thanks brion! [17:17:15] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2376753 (10yuvipanda) [17:17:26] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2059611 (10yuvipanda) [17:20:26] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Dafna3 was created, changed by Dafna3 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Dafna3 edit summary: Created page with "{{Tools Access Request |Justification=Don't know yet, just playing for now, I promise I'll be good |Completed=false |User Name=Dafna3 }}" [17:20:58] yuvipanda: fyi, the "staging" project has no instances and won't be coming back. The project itself can be /dev/null'd :) [17:21:51] :'( [17:22:37] Well, the long-term goal is to make beta more better [17:22:38] 06Labs, 10Labs-Infrastructure: Delete staging labs project - https://phabricator.wikimedia.org/T137726#2376776 (10yuvipanda) [17:22:38] :) [17:22:51] ostriches can you +1 there [17:23:29] {{done}} [17:23:48] thanks [17:25:44] andrewbogott if I delete a project, do I need to manually delete the domain associated with it? [17:25:52] yuvipanda: yes [17:26:04] how do I do that? [17:26:16] or should I just let you deal with it? :) [17:40:22] 06Labs, 10Labs-Infrastructure: Delete staging labs project - https://phabricator.wikimedia.org/T137726#2376890 (10yuvipanda) The domain staging.wmflabs.org needs to be deleted manually, according to @andrewbogott [18:19:31] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2377098 (10RobH) [21:49:37] 06Labs, 10Labs-Infrastructure: Disable diamond metrics for most labs projects temporarily - https://phabricator.wikimedia.org/T137753#2377699 (10yuvipanda) [21:57:41] 06Labs, 10Labs-Infrastructure, 06Operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2377737 (10Krenair) ```krenair@tools-bastion-03:~$ host 10.68.17.58 ;; Truncated, retrying in TCP mode. 58.17.68.10.in-addr.arpa domain name pointer ci-jessie-wikim... [21:59:46] (03CR) 10Jean-Frédéric: "Looks like Jenkins can process it fine. :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [22:24:06] 06Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing instance puppet config - https://phabricator.wikimedia.org/T91990#2377809 (10Andrew) First pass of a (still somewhat ugly) 'apply role' dialog. {F4162300} Questions: - How to distinguish between 'unset' and '' for params? - How to make...