START - Cookbook sre.hosts.decommission
ATTENTION: destructive action for 1 hosts: proton1001.eqiad.wmnet
Are you sure to proceed?
Type "done" to proceed
> done
Looking for matches in puppetmaster1001.eqiad.wmnet:/var/lib/git/operations/puppet
hieradata/common.yaml:- 10.64.0.200                  # kafka-main1001.eqiad.wmnet
hieradata/role/common/restbase/production.yaml:      listen_address: 10.64.0.209
Looking for matches in puppetmaster1001.eqiad.wmnet:/srv/private
Looking for matches in deploy1001.eqiad.wmnet:/srv/mediawiki-staging
Found match(es) in the Puppet or mediawiki-config repositories (see above), proceed anyway?
Type "done" to proceed
> done
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: ['proton1001.eqiad.wmnet']
Downtimed host on Icinga
Found Ganeti VM
Shutting down VM proton1001.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet
VM shutdown
Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
Sleeping for 20s to avoid race conditions...
Removed host proton1001.eqiad.wmnet from Debmonitor
Removed from DebMonitor
Removed from Puppet master and PuppetDB
Issuing Ganeti remove command, it can take up to 15 minutes...
Removing VM proton1001.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet. This may take a few minutes.
VM removed
Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
Generating the DNS records from Netbox data. It will take a couple of minutes.
Failed to run the sre.dns.netbox cookbook
Traceback (most recent call last):
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 296, in run
    dns_netbox_run(dns_netbox_args, spicerack)
  File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 61, in run
    results = netbox_host.run_sync(command, is_safe=True)
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 476, in run_sync
    batch_sleep=batch_sleep, is_safe=is_safe)
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 646, in _execute
    raise RemoteExecutionError(ret, 'Cumin execution failed')
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)
**Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required**
ERROR: some step failed, check the task updates.
Updated Phabricator task T255877
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)