[02:49:10] <wikibugs>	 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3523208 (10Marostegui) p:05Unbreak!>03Normal This graphs shows that the traffic got back to normal rate hours ago. What caused this...I don't know. But I think we can lower (if not close) this is...
[08:57:25] <wikibugs>	 10DBA: Implement cron-based mydumper backups on the dbstore role - https://phabricator.wikimedia.org/T169516#3523441 (10jcrespo) a:03jcrespo
[10:01:24] <wikibugs>	 10DBA, 10Patch-For-Review: Implement cron-based mydumper backups on the dbstore role - https://phabricator.wikimedia.org/T169516#3523529 (10jcrespo) Trying: ``` shard=s1 numthreads=8 mydumper --compress --host=localhost --threads=$numthreads --user=$(( whoami )) --socket=/run/mysqld/mysqld.$shard.sock --trigge...
[10:01:54] <wikibugs>	 10DBA, 10Patch-For-Review: Refactor puppet mariadb class to support multi-instance hosts - https://phabricator.wikimedia.org/T169514#3523530 (10jcrespo) a:03jcrespo
[12:18:26] <wikibugs>	 10DBA, 10Patch-For-Review: Implement cron-based mydumper backups on the dbstore role - https://phabricator.wikimedia.org/T169516#3523733 (10jcrespo) It took 3 hours to do the dump: ``` Started dump at: 2017-08-14 09:02:16 ... Finished dump at: 2017-08-14 12:04:49 ``` That is much worse than T162789#3238231 but...
[14:05:53] <wikibugs>	 10DBA, 10Patch-For-Review: Implement cron-based mydumper backups on the dbstore role - https://phabricator.wikimedia.org/T169516#3523921 (10jcrespo) s2 took less than 1 hour an a half, but its tables are much smaller, and we used 16 threads, and all replication threads were stopped: ``` Started dump at: 2017-0...
[21:07:25] <wikibugs>	 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3525085 (10Addshore) It looks like something just happened on db1082 again? at roughly 16:54 edits on wikidata dropped off again and there was a spike in the slave lag.    > <icinga-wm> IRC echo bot...
[22:07:46] <wikibugs>	 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3525153 (10Marostegui) I'm on a plane but the lag is probably due to the massive spike in UPDATES the master (db1063) had