[00:02:28] 06Labs: (Re-)Create Gitblit->Phabricator testing instance on Labs - https://phabricator.wikimedia.org/T142186#2529667 (10Danny_B) That //Labs// istance did not have anything to do with antimony or ytterbium. //git.wmflabs.org// was instance to test those redirects. IIANM it was basically just Apache and DNS conf... [01:42:51] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:08:36] 06Labs: (Re-)Create Gitblit->Phabricator testing instance on Labs - https://phabricator.wikimedia.org/T142186#2529800 (10Dzahn) It does have to do with phabricator since phabricator handles the redirects now and has a different setup from antimony. There are no .htacess files, it's in /modules/phabricator/templa... [02:10:50] 10Labs-project-Wikistats: move wikistats database to toolsdb - https://phabricator.wikimedia.org/T110391#2529805 (10Dzahn) 05Open>03declined [02:19:46] 06Labs, 10Phlogiston: Create new Phlogiston-01 instance - https://phabricator.wikimedia.org/T142277#2529844 (10JAufrecht) [02:22:51] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:27:00] 06Labs: (Re-)Create Gitblit->Phabricator testing instance on Labs - https://phabricator.wikimedia.org/T142186#2529876 (10Danny_B) I was mistaken - It was actually not .htaccess, but regular Apache config file (not that it would be a big difference for rewriting rules). Yes, I remember those conflicts. But they... [05:47:15] hi! not sure if this is the best place to ask, so trying my luck: I've registered my OAuth app on meta as 'owner-only'. Can I use it to test my OAuth consumer implementation before registering the full app? [05:49:23] Leloiandudu: partially [05:49:49] you can test the actual workings of the app but cannot test its authorization process [05:50:35] That's a shame :( How long does it usually takes for stewards to approve full app? [05:50:44] to do that, you can register the full app which will be sort of owner-only before approved [05:50:54] but with authorization process and everything [05:52:19] oh, you mean before it gets approved my account should still be able to auth via it? that's perfect! [06:04:50] also, do I get it right that I should proxy calls to mw api via my server (and sign them there) instead of issuing them from js directly? [06:22:30] tgr: ping :) [06:48:11] Leloiandudu: yes [06:48:26] cool, thank you very much [06:48:39] if you send it from JS, you need to publish the consumer secret [06:49:03] if you can avoid that, you should [06:55:15] PROBLEM - Puppet staleness on tools-proxy-01 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [08:44:05] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 06Operations: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#2530201 (10greg) [10:04:16] tgr: do you know what is the TTL of the access token? [10:07:06] can anyone get to Magnus's tool ... https://tools.wmflabs.org/wikidata-todo/creator_from_wikidata.php ? [10:22:13] sDrewthedoff: works for me [10:27:00] thanks Leloiandudu; after 10 minutes I get the page [10:27:11] hmm, closer to 20 min [11:52:30] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#2530374 (10Dvorapa) btw: Maybe better would be //draft// vs //done// (instead of //draft// vs //published//) if everything is public [14:15:40] 06Labs, 10Labs-Infrastructure: Default source group allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2530428 (10Andrew) I've run quite a lot of tests in the project 'testlabs,' generally using the host labvirt1014 which is a spare host and doesn't interfere with labs user... [14:32:57] 06Labs, 10Labs-Infrastructure: Default source group allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2530430 (10Andrew) I just ran this test on every single labvirt hosts. labvirt1001-1011 behave correctly. Only labvirt1012 through 1014 exhibit the bad 'no firewalling'... [15:37:44] Hi, is it known/expected that www.wmflabs.org and wmflabs.org aren't working? [15:39:11] jem: those don't go anywhere atm [15:39:16] http://tools.wmflabs.org/ [15:39:38] I know :), but I think they sould go somewhere [15:39:50] At least redirect to tools.wmflabs [15:48:59] to wikitech, rather, I think [15:54:47] 06Labs, 10Labs-Infrastructure: wmflabs.org does not resolve - https://phabricator.wikimedia.org/T38885#2530482 (10valhallasw) 05Resolved>03Open www.wmflabs.org is also broken by now. In general, we might want to consider having a wildcard for *.wmflabs.org with a 'This subdomain is not being serviced', no... [15:58:36] https://phabricator.wikimedia.org/T45580 was proposed too [16:27:38] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#2530492 (10yuvipanda) Yeah, I agree. I'm going to change the labels on it to avoid confusion. [16:45:38] valhallasw`cloud hi it seems the gerrit reviewer bot broke [16:45:40] again [16:46:15] maybe related to https://phabricator.wikimedia.org/T142155 [16:46:34] https://tools.wmflabs.org/gerrit-reviewer-bot/ [16:51:40] I did this https://gerrit.wikimedia.org/r/#/c/303368/ test patch [16:51:45] and it hasent added hashar [16:51:51] maybe it needs restarting [16:51:52] ? [16:56:07] 06Labs: (Re-)Create Gitblit->Phabricator testing instance on Labs - https://phabricator.wikimedia.org/T142186#2530505 (10Dzahn) Ok, i will set one up once we can create instances again. [18:03:36] paladox: somehow that changeset was never emailed [18:04:01] oh [18:04:08] it seemed to have failed for all of them [18:05:02] 'all of them'? [18:06:21] yep [18:06:45] AttributeError: 'NoneType' object has no attribute 'split' [18:06:45] CRITICAL: Waiting for 1 network thread(s) to finish. Press ctrl-c to abort [18:06:49] https://tools.wmflabs.org/gerrit-reviewer-bot/ [18:13:23] paladox: I'm concerned / annoyed that it errors out but then continues on in the next run :/ [18:13:36] Yep, i wonder what it could be [18:13:47] COuld it be because it needs restarting? [18:13:47] I think a different DELE behavior at gmail [18:13:53] oh [18:15:41] valhallasw`cloud does it show any errors in the log not shown on that page? [18:16:04] Maybe it is related to the mediawiki update this week? Maybe pywikibot needs updating on the gerrit reviewer bot? [18:16:14] paladox: please stop naming random causes [18:16:21] Oh sorry [18:17:58] PROBLEM - Host tools-docker-test-05 is DOWN: CRITICAL - Host Unreachable (10.68.22.123) [18:29:29] ....it's gmail. [18:29:31] damnit [18:37:23] ok. So the situation is as follows. In the old situation, gmail kept the email until a DELE POP command was sent. They changed this, and now it's deleted when it's RETR'd. [18:38:33] which is really weird, to say the least. [18:51:20] PROBLEM - Host tools-docker-test-04 is DOWN: CRITICAL - Host Unreachable (10.68.20.186) [18:52:53] PROBLEM - Host tools-docker-test-02 is DOWN: CRITICAL - Host Unreachable (10.68.20.76) [18:58:07] Host reboots coming up in 3 minutes. For more information see labs-l [18:58:20] when there is a DB problem, DBA was the phab project to add? [19:00:16] 06Labs, 10Tool-Labs, 10DBA: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530621 (10Giftpflanze) [19:11:58] PROBLEM - Host tools-worker-1019 is DOWN: CRITICAL - Host Unreachable (10.68.18.25) [19:12:13] ^ is all planned, is lal ok [19:12:32] PROBLEM - Host tools-worker-1008 is DOWN: CRITICAL - Host Unreachable (10.68.21.13) [19:12:44] PROBLEM - Host tools-docker-builder-01 is DOWN: CRITICAL - Host Unreachable (10.68.19.180) [19:12:50] PROBLEM - Host tools-worker-1009 is DOWN: CRITICAL - Host Unreachable (10.68.18.26) [19:13:02] maybe we can quiet shinken for that time? [19:13:15] yuvipanda: /mode +q shinken-wm!*@* ? [19:13:36] PROBLEM - Host tools-k8s-etcd-02 is DOWN: CRITICAL - Host Unreachable (10.68.18.64) [19:13:54] PROBLEM - Host tools-worker-1001 is DOWN: CRITICAL - Host Unreachable (10.68.23.55) [19:14:06] PROBLEM - Host tools-worker-1022 is DOWN: CRITICAL - Host Unreachable (10.68.21.130) [19:14:28] Luke081515 I've kicked it out for just now [19:15:28] ah, ok [19:15:32] thx [19:16:42] (03PS1) 10Merlijn van Deen: use TOP rather than REPR [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/303375 [19:16:55] (03CR) 10Merlijn van Deen: [C: 032] use TOP rather than REPR [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/303375 (owner: 10Merlijn van Deen) [19:18:24] (03Merged) 10jenkins-bot: use TOP rather than REPR [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/303375 (owner: 10Merlijn van Deen) [19:19:34] I have a question regarding grafana in labs, https://grafana-labs-admin.wikimedia.org/dashboard/db/ores-labs seems to be working but https://grafana-labs.wikimedia.org/dashboard/db/ores-labs says this dashboard doesn't exist and also grafana.wikimedia.org returns error for everything coming from labmon [19:19:55] (so grafana.wikimedia.org/dashboard/db/ores-labs is badly broken) [19:20:33] what should we do for grafana dashboard in labs [19:28:34] 06Labs, 10Labs-Infrastructure: www.wmflabs.org does not resolve - https://phabricator.wikimedia.org/T38885#2530656 (10Petrb) [19:29:11] 06Labs, 10Labs-Infrastructure, 10Graphite: Can't use wmflabs graphite datasource in grafana.wikimedia.org - https://phabricator.wikimedia.org/T141891#2530657 (10yuvipanda) I changed the URL to point directly to graphite-labs.wikimedia.org now. Try? I think @fgiunchedi would like us to move all labs stuff of... [19:29:17] 06Labs, 10Labs-Infrastructure: www.wmflabs.org does not resolve - https://phabricator.wikimedia.org/T38885#418242 (10Petrb) a:05RyanLane>03None [19:29:20] amir1 ^ [19:30:39] yuvipanda: nice [19:30:40] thanks [19:42:22] valhallasw`cloud but i use yahoo mail [20:09:49] (my default client does not allow me a confortabel topic change) [20:10:46] paladox: what does this have to do with your email...? [20:35:04] 10Tool-Labs-tools-Pageviews: Add option to view all redirects of a page to Massviews - https://phabricator.wikimedia.org/T141552#2530699 (10MusikAnimal) 05Open>03Resolved Done but as a separate app, at https://tools.wmflabs.org/redirectviews [20:59:51] valhallasw`cloud i thought you were saying it was a problem with gmail. [20:59:57] Yes. It is. [21:00:12] Oh, the bot uses gmail [21:00:15] Right. [21:00:39] but even though it is not emailing, shoulden it still be adding the reviewerers [21:01:13] the emails are the /input/ [21:02:01] oh [21:02:07] Why is gmail having problems? [21:03:59] in the old situation, when the bot crashed, the emails were kept until someone fixed the situation [21:04:09] oh [21:04:16] in the broken situation, all emails were deleted by gmail [21:04:59] oh [21:56:49] 06Labs, 10Tool-Labs: dplbot webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#2530757 (10yuvipanda) a:05yuvipanda>03None I've been checking it on and off for the last week or so and seems stable... [21:59:09] valhallasw`cloud: Do you know if some of the databases are lagging? [21:59:22] The wikidata db seems to be a bit behind [21:59:50] the disk seems full [22:00:45] Timestamp for most recent entry in recent changes table is 20160806184609. Hmm, that would explain it gifti [22:05:29] gifti: Do you know if we already have a bug in phabricator for this? [22:06:29] i have one for my special problem with it (can't drop table, T142305), but i guess no general one [22:06:29] T142305: Cannot drop table - https://phabricator.wikimedia.org/T142305 [22:06:40] thx, bot [22:09:54] multichill: http://tools.wmflabs.org/replag suggests so [22:10:29] gifti: why does your error suggest a full disk? [22:10:55] google does [22:11:18] Can you add that to your bug report? [22:11:23] and i couldn't update my other table, which supports that (though i don't know the error message there) [22:11:30] 06Labs, 10Tool-Labs, 10DBA: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530771 (10Multichill) [22:11:36] valhallasw`cloud: ^^ [22:11:54] ok [22:13:01] 06Labs, 10Tool-Labs, 10DBA: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530783 (10Giftpflanze) [22:13:38] 06Labs, 10Tool-Labs, 10DBA: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530621 (10Giftpflanze) [22:14:57] that's weird: for a ticket that i'm not subscribed to, first option it offers me is to unsubscribe … [22:15:02] and it does it … [22:15:20] oh, i was mentioned, maybe that was it [22:22:00] 06Labs, 10Tool-Labs, 10DBA: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530771 (10valhallasw) This seems to be only affecting c3 (labsdb1003). c2 (labsdb1002) is still offline. ``` valhallasw@tools-bastion-03:~$ echo "SELECT * FROM heartbeat_p.heartbeat... [22:22:26] multichill: as a workaround, switch to c1.labsdb [22:22:58] Yeah, already did that :-) [22:23:26] * valhallasw`cloud is off to bed [22:24:19] Good plan [22:26:49] 06Labs, 10Labs-Infrastructure: Set up some sort of web pages at wmflabs.org or www.wmflabs.org - https://phabricator.wikimedia.org/T38885#2530790 (10AlexMonk-WMF) [22:39:04] 06Labs, 10Labs-Infrastructure: Set up some sort of web pages at wmflabs.org or www.wmflabs.org - https://phabricator.wikimedia.org/T38885#2530802 (10AlexMonk-WMF) @Andrew, @YuviPanda: How about I CNAME these domains to proxy-eqiad.wmflabs.org (novaproxy-01 in project-proxy), and make the dynamicproxy web serve... [22:47:56] andrewbogott: are you currently available? (nothing urgent) [22:48:27] yuvipanda: or you instead? [22:49:00] there's a page for labsdb1003 so I'm looking at it right now [22:50:29] ok [22:51:16] I just want to know, if I can request a extesion of my current quota for one of my projects. the reason is, that I want to rebuild an instance, but need to setup a new first, to switchover without an interruption of services [22:51:25] would be a medium instance [22:53:59] yuvipanda: FYI, if you have some time ^ [22:55:33] Luke081515 there's a process for it now https://phabricator.wikimedia.org/T140904 [22:56:22] ah, ok :) [23:04:42] 06Labs: Request increased quota for rcm labs project - https://phabricator.wikimedia.org/T142311#2530827 (10Luke081515) [23:04:43] yuvipanda: done now [23:05:13] 06Labs: Request increased quota for rcm labs project - https://phabricator.wikimedia.org/T142311#2530841 (10Luke081515) p:05Triage>03Low [23:06:30] yuvipanda: hi! what's the TTL of OAuth access token? [23:09:28] Leloiandudu: there isn't any [23:10:40] tgr: not sure if it is good from security standpoint but that definitely simplifies my code. thanks again! [23:20:53] 06Labs, 10Tool-Labs, 10DBA: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530855 (10yuvipanda) labsdb1003 had crashed, @jcrespo rescued it and I see replag go down now. [23:25:26] 06Labs, 10Tool-Labs: dplbot webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#2530861 (10Gorthian) It's down right now. It has been intermittently down and back up since I last posted here. [23:29:49] 06Labs, 10Tool-Labs: dplbot webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#2530862 (10yuvipanda) strange, I just checked it and it's up... [23:32:23] 06Labs, 10Tool-Labs, 10DBA: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530868 (10jcrespo) Both labsdb1001 and labsdb1003 crashed this week due to excessive memory pressure (3 GB swap file). Until we determine if there is a tools or tools using more reso... [23:45:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:50:06] 06Labs, 10Tool-Labs, 10DBA: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530889 (10Giftpflanze) 05Open>03Resolved a:03Giftpflanze Seems to work again. [23:54:25] 06Labs, 10Tool-Labs: dplbot webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#2530900 (10russblau) I've had to restart it manually at least once a day for the past several days, although not today (so far).