[00:00:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [00:18:55] Krenair: chasemp: refactored https://wikitech.wikimedia.org/wiki/Add_a_wiki a bit [00:23:23] Krinkle, "confirm? is it okay to sync the config to app servers before the db exists?" [00:23:25] I wouldn't [00:23:37] anything using foreachwiki and co. will break [00:23:50] https://wikitech.wikimedia.org/w/index.php?title=Add_a_wiki&diff=prev&oldid=1069880 [00:23:56] yeah [00:24:27] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [00:25:26] I'll probably check this in detail next time we use it, which will probably be pretty soon, there's another wiki in the process [00:31:40] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [03:03:57] 06Labs, 10Horizon, 07Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#2843483 (10Andrew) [03:03:59] 06Labs, 10Horizon, 13Patch-For-Review, 07Upstream: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2843480 (10Andrew) 05stalled>03Resolved a:03Andrew I've fixed the upstream bug in future versions, hacked a fix locally, and also added the 'remember me' checkbox. [04:09:02] 10Tool-Labs-tools-Pageviews: Implement a monthly granularity view for Pageviews - https://phabricator.wikimedia.org/T151373#2843517 (10MusikAnimal) Will wait for T139934 instead of implementing a clientside solution [04:09:15] 10Tool-Labs-tools-Pageviews: Implement a monthly granularity view for Pageviews - https://phabricator.wikimedia.org/T151373#2843519 (10MusikAnimal) [05:01:50] PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:02:00] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:02:24] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:02:44] that's me testing, should be fine soon [05:02:59] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:03:01] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:03:07] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:03:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:03:31] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:03:57] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:04:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:04:49] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:04:53] PROBLEM - Puppet run on tools-elastic-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:05:23] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:05:39] PROBLEM - Puppet run on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:05:47] PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:07:18] PROBLEM - Puppet run on tools-exec-1206 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:07:30] PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:07:40] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:17:21] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:38:29] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [05:40:21] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:41:49] RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [05:41:59] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [05:42:28] RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:42:58] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [05:43:00] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [05:43:06] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [05:43:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [05:43:54] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:06] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:46] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:50] RECOVERY - Puppet run on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:45:41] RECOVERY - Puppet run on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [05:45:50] RECOVERY - Puppet run on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [05:47:19] RECOVERY - Puppet run on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [05:47:37] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [13:49:28] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2843924 (10yuvipanda) I'm going to write a script called `maintain-dbusers` with the followi... [16:55:10] 06Labs, 10Labs-Infrastructure, 07Puppet: realm.pp: "Data retrieved from Toolsbeta is String not Hash" if not defined in Hiera - https://phabricator.wikimedia.org/T152142#2844202 (10scfc) This happens with standalone puppetmasters as well, and `/var/log/syslog` then says: ``` Dec 3 16:29:41 toolsbeta-puppet... [18:13:11] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2844373 (10jcrespo) Replication had broken because events from both sanitarium, production slaves and labs were running there... [20:28:46] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2844764 (10Marostegui) >>! In T152194#2844373, @jcrespo wrote: > Replication had broken because events from both sanitarium,... [21:29:58] Change on 12www.mediawiki.org a page OAuth/Owner-only consumers was modified, changed by Tgr (WMF) link https://www.mediawiki.org/w/index.php?diff=2300763 edit summary: [22:11:38] 06Labs, 10Labs-Infrastructure, 07Puppet: mwyaml chokes on existing, but empty Hiera: pages on wikitech - https://phabricator.wikimedia.org/T152142#2844932 (10scfc) p:05Triage>03High a:03scfc [22:21:31] 06Labs, 10Labs-Infrastructure, 07Puppet: mwyaml chokes on existing, but empty Hiera: pages on wikitech - https://phabricator.wikimedia.org/T152142#2844937 (10scfc) The issue is caused by existing, but empty `Hiera:` pages and a very misleading (:-)) difference between code and error message in `modules/wmfli... [22:23:23] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [22:23:40] Change on 12www.mediawiki.org a page OAuth/Owner-only consumers was modified, changed by Tgr (WMF) link https://www.mediawiki.org/w/index.php?diff=2300770 edit summary: /* PHP */ [22:50:36] 10Labs-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2844997 (10scfc) [22:51:10] !log wikimania-support Tried to build a wikimania-scholarships-01 instance, but it seems to have had a fatal Puppet error on initial provisioning. Tried reboot but that doesn't seem to have helped. [22:51:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikimania-support/SAL [22:52:34] 06Labs, 10Labs-Infrastructure, 07Puppet: mwyaml chokes on existing, but empty Hiera: pages on wikitech - https://phabricator.wikimedia.org/T152142#2844998 (10scfc) (AFAIUI, after deploying the change, the Labs puppetmaster needs to be restarted (`service apache2 restart`) because Puppet/Ruby does not reload... [23:18:34] 10Labs-project-Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2845005 (10Paladox) The main phabricator prod role works now :)