[00:34:46] 06Labs, 10Tool-Labs: Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#2967407 (10bd808) That seems like a good idea. We could move the actual script into the labs/toollabs deb with `become`, `jsub`, etc with that bit of config moved outside the script. [01:08:25] 06Labs, 10DBA, 10wikitech.wikimedia.org: SemanticMediaWiki tries to create temporary tables, but can't as wikiuser is restricted - https://phabricator.wikimedia.org/T110981#1591340 (10jcrespo) If this was me, I would close it as won't fix- this fails because the user doesn't have permissions to do `CREATE TE... [01:54:08] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10Revent) I was asked to mention this here, although I'm unsure if it's actually a 'replication' issue. https://quarry.wmflabs.org/query/14916 reports 35 entries in the transcode table, all for files that were uplo... [02:47:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:00:11] 06Tool-Labs-standards-committee: Create mailing list for Tool-Labs-standards-committee - https://phabricator.wikimedia.org/T156218#2967762 (10Quiddity) [03:01:22] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2963376 (10Quiddity) * Email: I think just a private mailing list, might be sufficient to start with? I've started a basic task at {T156218} We j... [03:04:37] 06Tool-Labs-standards-committee: Create mailing list for Tool-Labs-standards-committee - https://phabricator.wikimedia.org/T156218#2967794 (10Huji) [03:14:56] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2967820 (10MZMcBride) Another request for this data for `frwiki_p`: . [03:38:04] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:09:30] 10Tool-Labs-tools-Other, 10Possible-Tech-Projects: Fix TreeViews to provide pageviews statistics for all articles of any wikiproject etc. - https://phabricator.wikimedia.org/T56184#2967897 (10Upeksha1996) a:03Upeksha1996 [04:45:25] 10Tool-Labs-tools-Other, 10Possible-Tech-Projects: Fix TreeViews to provide pageviews statistics for all articles of any wikiproject etc. - https://phabricator.wikimedia.org/T56184#2967918 (10Doc_James) Looking forwards to seeing this solved :-) [06:36:51] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2963376 (10eranroz) * +1 for tool-labs-standards-committee@lists.wikimedia.org * Meetings: +1 to a regular meeting via hangout (~quarterly) [06:43:06] RECOVERY - Free space - all mounts on tools-exec-1221 is OK: OK: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [06:55:58] 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2968016 (10Marostegui) [07:04:41] 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2968020 (10Marostegui) For the record and tracking purposes: after lots of hours and hassle we were able to switch db1095's (new sanitarium) master from db1052... [07:08:02] 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2961118 (10Marostegui) [07:48:00] !log video repooling encoding02 [07:48:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [09:53:16] 06Tool-Labs-standards-committee, 10Wikimedia-Mailing-lists: Create mailing list for Tool-Labs-standards-committee - https://phabricator.wikimedia.org/T156218#2968275 (10zhuyifei1999) [09:54:20] 06Tool-Labs-standards-committee: Create mailing list for Tool-Labs-standards-committee - https://phabricator.wikimedia.org/T156218#2968277 (10zhuyifei1999) [10:21:22] !log video `$ sudo apt update && sudo apt dist-upgrade` on all hosts [10:21:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [10:26:00] 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2968376 (10Marostegui) >>! In T155902#2965899, @Tb wrote: > Great thanks. although I missed one in the list above; can you grant all to s51111 on p50380g50491_inconsistent_redirects on s1.la... [10:29:47] 06Labs: Lower quotas to current usage for dwl (when ready) - https://phabricator.wikimedia.org/T152456#2968380 (10Giftpflanze) So, we deleted the old instace before creating the new one anyway because we wanted to keep the name. You can now readjust the quota. [10:47:10] 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup, 15User-bd808: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2968416 (10WMDE-leszek) I've also come across the exact same issue when trying to install Mediawiki Vagrant in the newly crea... [11:07:19] 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2968453 (10faidon) p:05High>03Unbreak! [11:10:22] 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2968455 (10faidon) The switch rebooted again overnight (Jan 25 01:16 UTC). We are going to proceed with a replacement as soon as the DBA work (T155999) is done. Setting priority to... [11:14:03] 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2968471 (10Marostegui) >>! In T155875#2968455, @faidon wrote: > The switch rebooted again overnight (Jan 25 01:16 UTC). We are going to proceed with a replacement as soon as the DB... [11:28:52] 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2968484 (10Tb) 05Open>03Resolved [11:49:08] PROBLEM - Free space - all mounts on tools-exec-1221 is CRITICAL: CRITICAL: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-1221.diskspace.root.byte_percentfree (<11.11%) [12:00:33] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2968516 (10jcrespo) This is actually done on all wikis- but for some reason, there is a bug and the views have not been regenerated. ``` $ mysql --skip-ssl -h labsdb1001.eqiad.... [12:48:52] I've I get a 502 for http://korma.wmflabs.org/ is there anything I myself can do? Probably not? :-/ [12:52:27] andre__: poke the admins :P [13:29:00] (03PS1) 10Jean-Frédéric: Change ID type to varchar for cm_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/334082 (https://phabricator.wikimedia.org/T156139) [13:30:01] !log tools.heritage Deploy latest from Git master: 8c75342 [13:30:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [13:32:48] (03CR) 10Jean-Frédéric: "Harvesting tested in Docker, all good." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/334082 (https://phabricator.wikimedia.org/T156139) (owner: 10Jean-Frédéric) [13:40:01] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10Aklapper) [13:40:11] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10Aklapper) p:05Triage>03High [14:43:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:43:58] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10Paladox) @Aklapper korma is deprecated, it has been replaced by https://wikimedia.biterg.io/app/kibana#/dashboard/Overview See https://www.mediawiki.org/wiki/... [14:45:06] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2968832 (10chasemp) @jcrespo, for now the runs are not automated. I think this is good to go now and took: > maintain-views --all-databases --table watchlist_count --replace-all [14:46:45] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2968834 (10jcrespo) @chasemp Thank you very much! I think I can run that on my own, at least for frwiki. [14:51:56] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2968839 (10jcrespo) Oh, sorry, I misunderstood it. You run it already. @MZMcBride can you test it, even if T59617#2893932 is still pending? ``` $ mysql --skip-ssl frwiki_p -e "... [14:56:20] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968863 (10Aklapper) @Paladox: No. "korma.wmflabs.org will be deprecated" is not "korma.wmflabs.org has been deprecated". Please read carefully. [15:17:18] 06Labs, 10Labs-Infrastructure, 10Deployment-Systems, 10Salt, and 3 others: Can not use git-deploy from tin.eqiad.wmnet to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T111925#1619993 (10mschwarzer) @hashar Hi. I'm having the same problem, when using Puppet to install **role::elasticsea... [15:39:04] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:58:16] 06Labs, 10Labs-Infrastructure, 10Deployment-Systems, 10Salt, and 3 others: Can not use git-deploy from tin.eqiad.wmnet to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T111925#2969157 (10hashar) context ====== Nodepool is a python software running on labnodepool1001.eqiad.wmnet. The sof... [16:23:30] 10Striker, 15User-bd808: Deploy striker on labtestweb2001 - https://phabricator.wikimedia.org/T156276#2969237 (10Andrew) [16:23:56] 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup, 15User-bd808: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2969253 (10bd808) @Ladsgroup @WMDE-leszek before we try to get too fancy in debugging this, could either or both of you try r... [16:26:45] 06Labs: Revert increased quota for services-test labs project - https://phabricator.wikimedia.org/T153711#2969263 (10Eevans) I have terminated the m1.xlarge instance I created for heap analysis, and you can now revert the quota increase. I realize that I kept this for longer than I originally indicated, sorry a... [16:31:59] 10Striker, 15User-bd808: Deploy striker on labtestweb2001 - https://phabricator.wikimedia.org/T156276#2969269 (10bd808) The big trick here will be figuring out how to separate from scap3 for the deploy step. The existing Puppet setup uses `service::uwsgi` which automatically sets up scap3 deployment. It looks... [16:36:39] 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup, 15User-bd808: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2969273 (10WMDE-leszek) @bd808 I've done a "soft reboot" of the instance in Horizon but when bringing the host VM up I am sti... [16:36:43] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10Krenair) @Revent: As the production database servers return the same result, this has nothing to do with labs replication. [16:37:54] andre__: I'll look in a minute [16:39:09] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2969276 (10jcrespo) @Revent then I missunderstood you, we should file a proper, standalone bug. [16:55:32] 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2969340 (10Marostegui) [16:58:18] andre__: do you know which labs project this is? [16:58:41] 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2969364 (10Cmjohnson) @faidon new switch has been installed. Also added an uplink module. The switch is accessible via mgmt [17:14:32] (03CR) 10Multichill: [C: 032] Change ID type to varchar for cm_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/334082 (https://phabricator.wikimedia.org/T156139) (owner: 10Jean-Frédéric) [17:17:05] (03Merged) 10jenkins-bot: Change ID type to varchar for cm_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/334082 (https://phabricator.wikimedia.org/T156139) (owner: 10Jean-Frédéric) [17:18:32] (03CR) 10jenkins-bot: Change ID type to varchar for cm_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/334082 (https://phabricator.wikimedia.org/T156139) (owner: 10Jean-Frédéric) [17:27:39] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2969456 (10chasemp) >>! In T143349#2962882, @chasemp wrote: >>>! In T143349#2946955, @Acs wrote: >>>>! In T143349#2946796, @chasemp wrote: >>> @Qgil and @acs do you know if the instance... [17:28:41] andre__: my impression was quim and acs said to remove the instance but I'm not sure they knew exactly what teh setup was there or the the implicatoins of the question https://phabricator.wikimedia.org/T143349#2969456 [17:28:53] this is really on all 3 of us for missed messages [17:29:36] whoops :-/ [17:30:42] it's not superurgent to get it up but would be very nice to get it back, until https://wikimedia.biterg.io/ (to deprecate it) has feature-parity [17:30:52] ....which isn't yet the case until https://phabricator.wikimedia.org/T137997 has open dependencies [17:31:28] Guess I should have taken a closer look at that task too, sorry :-/ [17:31:52] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2969480 (10Aklapper) potentially due to T143349#2969456 [17:37:40] andre__: yeah I'm sure who owned korma, or if it's worth it considering the idea is to move to that already. Open to suggestions on next course of action [17:38:42] I meant not sure who owned it [17:39:22] chasemp: basically Bitergia (WMF's contractor for metrics about the technical community). Yeah let me think about what to do in the next days. Meh [17:40:03] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2969538 (10Superyetkin) I am getting timeout errors on `labsdb-web.eqiad.wmnet` . [17:41:42] andre__: really sorry man, let me know how I can help. [17:43:08] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10madhuvishy) @Aklapper Hi! Yes, unfortunately it looks like the instance that hosts korma.wmflabs.org got deleted as in T143349#2969456. Do let us know if we ca... [17:44:15] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2257889 (10chasemp) >>! In T134203#2969538, @Superyetkin wrote: > I am getting timeout errors on `labsdb-web.eqiad.wmnet` . Can you give us some more details? From where, etc? [17:49:43] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2969585 (10Superyetkin) [[http://tools.wmflabs.org/superyetkin/test.php | Here]] is a page where the connection method (mysql_connect) is being called. [17:50:41] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2969587 (10jcrespo) also, does this mean labsdb-analytics.eqiad.wmnet works for you? [17:55:09] 06Labs, 10Tool-Labs, 10DBA: Labs users reporting timouts when connecting to labsdb-web.eqiad.wmnet - https://phabricator.wikimedia.org/T156285#2969602 (10jcrespo) [17:55:34] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2257889 (10jcrespo) Let's move to a different ticket: T156285 [18:06:28] 06Labs, 10Labs-Infrastructure, 10Deployment-Systems, 10Salt, and 3 others: Can not use git-deploy from tin.eqiad.wmnet to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T111925#1619993 (10greg) To be clear, git-deploy/trebuchet is **deprecated** as of March 11th, 2016. https://lists.wikime... [18:24:55] 06Labs, 10Tool-Labs, 10DBA: Labs users reporting timouts when connecting to labsdb-web.eqiad.wmnet - https://phabricator.wikimedia.org/T156285#2969750 (10chasemp) I'll look into this today if I can or very shortly thereafter [18:24:58] 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2969751 (10bd808) [18:29:45] 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2969769 (10bd808) [18:40:39] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2969823 (10MZMcBride) >>! In T59617#2968839, @jcrespo wrote: > @MZMcBride can you test it, even if T59617#2893932 is still pending? Yay, it works! Output from `frwiki_p`: !log wikilabels 34c2b0c is going to staging [18:44:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [18:56:00] Looks good on every stage [18:56:05] going prod [18:56:22] !log wikilabels 34c2b0c is going to prod [18:56:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [18:56:35] 06Labs, 10Tool-Labs, 10DBA: Labs users reporting timouts when connecting to labsdb-web.eqiad.wmnet - https://phabricator.wikimedia.org/T156285#2969602 (10Krenair) I'm guessing this will be #netops filtering between production and labs, though I'm surprised this only came up after actual users were given the... [18:58:07] !log tools.heritage Deploy latest from Git master: 0810246 (T156139) [18:58:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [18:58:11] T156139: Investigate cm_(fr) harvesting (Cameroon monuments in French) - https://phabricator.wikimedia.org/T156139 [19:07:51] 10Tool-Labs-tools-Other: Coordinate links should remember user preference for the mapping service and not ask to choose every time - https://phabricator.wikimedia.org/T153211#2969926 (10Yurivict) Clicking "Link to other maps" isn't persistent there. So it is going to require many clicks to go to non-OSM map just... [19:23:46] 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2969962 (10bd808) [19:24:10] 10Striker, 06Community-Tech-Tool-Labs, 05Goal, 13Patch-For-Review, 15User-bd808: Create Wikitech/LDAP accounts via a new user friendly guided workflow - https://phabricator.wikimedia.org/T144710#2969967 (10bd808) 05Open>03Resolved [19:24:24] 10Striker, 06Community-Tech-Tool-Labs, 05Goal, 13Patch-For-Review, 15User-bd808: Create Wikitech/LDAP accounts via a new user friendly guided workflow - https://phabricator.wikimedia.org/T144710#2607979 (10bd808) [19:24:26] 10Striker, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: Striker should respect TitleBlacklist bans on new account names - https://phabricator.wikimedia.org/T147024#2969971 (10bd808) 05Open>03Resolved [19:24:34] 10Striker, 13Patch-For-Review, 15User-bd808: Check for 2FA protection and enforce validation of 2FA tokens - https://phabricator.wikimedia.org/T144712#2969973 (10bd808) 05Open>03Resolved a:03bd808 [19:24:45] 10Striker, 13Patch-For-Review, 15User-bd808: Allow management of LDAP SSH keys - https://phabricator.wikimedia.org/T144711#2969980 (10bd808) 05Open>03Resolved a:03bd808 [19:25:04] 10Striker, 13Patch-For-Review, 15User-bd808: Allow changing LDAP password from Striker - https://phabricator.wikimedia.org/T153935#2969981 (10bd808) 05Open>03Resolved a:03bd808 [19:35:40] 06Labs, 10Tool-Labs, 10PageImages, 06Reading-Web-Backlog, and 2 others: Data disappeared from labs replica in cswiki_p.page_props - https://phabricator.wikimedia.org/T153888#2970047 (10Jdlrobson) 05Open>03Resolved a:03Jdlrobson Please reopen if you have any further questions. [19:35:44] Error: Could not request certificate: Connection refused - connect(2) [19:36:02] Getting that when creating a new instance ^ [19:36:28] puppet appears to be failing to request SSL keys for the instance. [19:37:50] it also does not come with an instance ID [19:38:39] one of them was launched over 20 minutes ago [19:44:07] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2970072 (10Paladox) @madhuvishy Hi, im not sure if this will help, but it points to the source code here https://github.com/Bitergia/mediawiki-dashboard per https://www.m... [19:46:08] Just submitted a proposal for an OAuth consumer - can I modify the callback url? [19:46:20] bd808: just fyi re. T143349 I am attempting to start to migrate for UTRS, with some hiccups [19:46:21] T143349: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349 [19:57:42] samtar: you have to file a new consumer request to change the callback url. Don't forget to increment the version number [19:58:09] bd808: should I wait until the previous request is approved? [19:58:35] samtar: no, but I can cancel the old one for you if you'd like [19:59:05] samtar: this one? https://meta.wikimedia.org/wiki/Special:OAuthManageConsumers/28dd8667e0b7778945a8c27d767cd7a7 [19:59:28] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs: Make a nag system to email maintainers of tools still running on precise gird hosts - https://phabricator.wikimedia.org/T149214#2745525 (10madhuvishy) I used this and a bit more to write a script that maps users to precise tools, and vice versa - https://phabr... [20:00:08] AmandaNP: yuck. would you like me to look at the instance logs and try to see if its something that can be recovered from? [20:00:09] bd808: unauthorised so guessing not? The consumer name is `LTA Knowledgebase [1.0]` if that helps? [20:00:27] samtar: heh. sorry I linked to the admin view of the request [20:01:14] I would guess that in your new request you should a) use https://... and b) check the "Allow consumer to specify a callback" checkbox [20:01:28] bd808: yes please. I also created a second instance, showing the same issue. utrs-live and utrs-database [20:02:41] bd808: sounds like a good idea.. if you do make it disappear could you pop a note at https://meta.wikimedia.org/wiki/Steward_requests/Miscellaneous#OAuth_application so MarcoAurelio doesn't get confused? ^^ [20:03:05] will the app show the emails of other users? [20:05:06] AmandaNP: ugh. the initial puppet run broke before it even setup root access. [20:05:37] -_- [20:06:13] bd808: Not publicly no, but it will use the emails to send notifications if required - this will be clearly stated on the OAuth sign-in page (currently an email address is required to request an account) [20:06:33] is it something i'm screwing up or is it an operations issue [20:06:37] samtar: *nod* [20:07:08] AmandaNP: I would guess broadly that this is our problem and not yours. [20:07:34] AmandaNP: could you open a phabricator task documenting the instance failures? [20:07:48] sure. Whole log or the failure points? [20:08:05] putting the log that you can see in won't hurt [20:09:50] samtar: I'm looking at https://tools.wmflabs.org/lta/request/. Will that be replaced with OAuth once you get it working? [20:10:02] bd808: indeed [20:10:17] samtar: cool deal [20:10:46] it would be best too if instead of storing the emails locally you used the api and the oauth token to fetch them when needed. [20:11:15] that way the end user can block you from having their address by revoking the oauth grant [20:12:02] to do that fetch you would just store the oauth token for each user as they sign up locally and use it with the api when needed [20:13:33] oauth is like the standard now adays [20:13:56] Ah I see bd808 - OAuth is entirely new to me I'm afraid, I've just been pestered into implementing it as username:password is "old hat" [20:14:20] ultimately all I'd need in the user table then is a list of usernames which are approved? [20:14:33] samtar i think theres an api you can use for oauth and wmf stuff [20:14:44] we need some better tutorials on how to integrate it nicely [20:15:14] https://www.mediawiki.org/wiki/OAuth/For_Developers has been helpful enough :) it was working okay until I realised the callback was going to be an issue :P [20:17:10] 06Labs, 10Labs-Infrastructure: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970184 (10DeltaQuad) [20:18:18] samtar: basically what I'm suggesting is that you store the token that you get from the Special:OAuth/token in your user table. Then you can use this token at any time in the future to call the identity end point and retrieve the user's email address [20:18:45] and the user can decide that they don't want you to have access anymore and revoke the token which gives you that access [20:18:49] bd808: is T148929 actually already calling out the problem? [20:18:50] T148929: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929 [20:19:27] bd808 are you tools server admin? [20:19:36] nevermind [20:19:37] AmandaNP: did you enable the stand-alone puppetmaster role on those servers? [20:19:47] my brain is thinking in wrong places [20:20:31] no clue what that even is. I don't think that option even presents itself when I create an instance [20:21:19] AmandaNP: :) cool. you did not then. That bug is probably similar in log messages but different in cause [20:21:32] fair enough [20:23:55] 06Labs, 10Labs-Infrastructure, 07Puppet: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970242 (10DeltaQuad) [20:24:51] bd808: whilst you're about, and I know you're super busy - but I've now set the consumer so I can specify the callback. Is that specified in `oauth_callback`? Just the example here has the comment `Must be oob for MWOAuth` and I don't quite understand [20:33:38] bd808: woo! Yup works :) [20:33:43] sweet [20:34:09] Thanks for your help! [20:34:30] That example app was from the very first OAuth deploy and I think we didn't support the prefix option then [20:34:49] Somebody should send anomie a patch to update that part of it [20:41:11] 06Labs, 10Labs-Infrastructure, 07Puppet: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970184 (10bd808) The key bit here seems to be `Could not request certificate: Connection refused` when trying to talk to the Puppetmaster. The `+ sed -i s/_MASTER_//g /etc/puppet/puppet.c... [20:42:18] madhuvishy: do you know how the first boot stuff is supposed to work to bootstrap the Puppetmaster connection? -- https://phabricator.wikimedia.org/T156297 [20:44:41] hostname: Name or service not known [20:44:49] looks familiar [20:45:31] The sed replacing _MASTER_ with nothing seems suspicious [20:45:53] right [20:46:20] i'm just grabbing lunch, i can look in a bit - but i don't know much [20:46:20] I just booted a new instance and it says "+ sed -i s/_MASTER_/labs-puppetmaster-eqiad.wikimedia.org/g /etc/puppet/puppet.conf" [20:47:06] madhuvishy: cool. looks like chasemp is back. maybe he can point me in the right direction [20:47:17] okay, i'll check in in a bit [20:48:22] bd808: this used to happen (hostname: name or service not found) when we had the designate rdns issue [20:48:32] with entries not being deleted in the designate db [20:48:40] i thought it was fixed [20:49:56] bd808: well, I believe when it comes up the first action is to run /root/firstboot.sh and 'sed -i s/_MASTER_//g /etc/puppet/puppet.conf' is indeed uh suspicious [20:50:06] 06Labs, 10Labs-Infrastructure, 07Puppet: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970313 (10bd808) I tried to reproduce by creating a new instance and see a big difference in that initial setup: ``` + project=mediawiki-vagrant ++ curl http://169.254.169.254/1.0/meta-da... [20:50:15] a project set up for a standalone master but something is array post puppetmaster::standalone? [20:50:19] chasemp: yeah. see that comment ^^ [20:50:33] I'm wrapped up in something else for a minute here, what project did yours come up successfully in? [20:50:43] mediawiki-vagrant [20:50:47] ok [20:50:53] and failure is in utrs [20:51:05] let's try another project, I'm wondering if this isn't project specific [20:51:11] I'll circle back in a few [20:52:33] !log wikilabels sudo -u www-data /srv/wikilabels/venv/bin/wikilabels new_campaign enwiki "Discussion quality" toxicity DiffToPrevious 5 50 [20:52:33] the old designate issue surfaced in a similar way as in bad puppet cert issues but this is slightly different [20:52:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [20:52:47] I'm not sure exactly what condition would cause the master to be set empty atm [20:52:52] !log wikilabels less ~/toxicity_input.json | sudo -u www-data ../venv/bin/wikilabels task_inserts 47 [20:52:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [20:57:31] 06Labs, 10Tool-Labs, 10DBA: Labs users reporting timouts when connecting to labsdb-web.eqiad.wmnet - https://phabricator.wikimedia.org/T156285#2970337 (10chasemp) 05Open>03Resolved a:03chasemp @jcrespo should be sorted now [21:02:27] bd808: ok so I had an instance come up in toolsbeta [21:03:28] 06Labs, 10Labs-Infrastructure, 07Puppet: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970184 (10chasemp) Does project use its own puppetmaster? [21:03:47] chasemp: cool. seems to be somehow localized then. I haven't tried to poke into how the utrs project is setup [21:04:34] chasemp: AmandaNP didn't know what I was talking about when I asked that, but it could be something someone else setup as project global config [21:04:54] bd808: one thing is andrew sent me a link yesterday or today I have lost local history on in -admin saying something about new instances in NFS enabled projects has a known issue I'm not sure if related, need to search my irc archive [21:05:05] should not result in this puppet issue tho [21:05:39] ah ok bd808 probably not then but hm [21:14:04] 10Tool-Labs-tools-LTA-Knowledgebase: Migrate to OAuth - https://phabricator.wikimedia.org/T155841#2970436 (10DatGuy) a:05DatGuy>03Samtar [21:20:47] !log ores deployed ores-wmflabs-deploy:0f90516 [21:20:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:20:58] We now have a draftquality model! [21:21:04] That's one wishlist item down :D [21:21:31] * halfak prepares to email qgil the good news [21:22:22] 06Labs, 10Tool-Labs: Unable to connect to new database servers - https://phabricator.wikimedia.org/T156307#2970447 (10russblau) [21:34:03] 06Labs, 10Tool-Labs: Unable to connect to new database servers - https://phabricator.wikimedia.org/T156307#2970447 (10chasemp) @russblau, can you try again please? [22:06:32] 06Labs, 10Tool-Labs: Unable to connect to new database servers - https://phabricator.wikimedia.org/T156307#2970611 (10chasemp) 05Open>03Resolved a:03chasemp let me know if not working still thanks :) [22:11:34] 06Labs, 06Operations, 13Patch-For-Review: Set up monitoring for secondary labstore HA cluster - https://phabricator.wikimedia.org/T144633#2970617 (10chasemp) @Madhuvishy satisfied we can close? [22:11:53] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: Migrate misc to secondary labstore HA cluster - https://phabricator.wikimedia.org/T154336#2970618 (10chasemp) @Madhuvishy do you think we can close this now? [22:13:56] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2970629 (10madhuvishy) [22:13:58] 06Labs, 06Operations, 13Patch-For-Review: Set up monitoring for secondary labstore HA cluster - https://phabricator.wikimedia.org/T144633#2970627 (10madhuvishy) 05Open>03Resolved @chasemp Yup, closing. [22:18:16] (03Abandoned) 10Paladox: Add the phabricator and phab-* project o #wikimedia-releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 (owner: 10Paladox) [22:18:40] (03PS3) 10Paladox: Add project phab-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 [22:19:30] (03PS3) 10Paladox: Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 [22:20:55] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2970654 (10chasemp) [22:20:58] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2970653 (10chasemp) [22:28:46] 06Labs, 10Labs-Infrastructure, 07Puppet: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970675 (10DeltaQuad) As mentioned on IRC, I have no clue. The option was not presented in the setup options as far as I know. [23:05:39] 06Labs, 10Labs-Infrastructure, 07Puppet: Puppet failure on instance creation - https://phabricator.wikimedia.org/T156297#2970732 (10chasemp) @DeltaQuad ok thank you, @Andrew can you look at this when you get a second? I'm not sure what's going on at the moment in this project. [23:31:30] 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup, 15User-bd808: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2970818 (10bd808) I can recreate this problem on a fresh host. Next I'll try to see if it is fixed by https://gerrit.wikimedi... [23:32:04] 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup, 15User-bd808: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2970820 (10bd808)