When I tried to do a Qualys vulnerability scan of a Fedora 29 system today, things didn’t go too smoothly.
First, I downloaded and installed the Qualys Cloud Agent – so far, so good.
The next step was to run an included shell script to activate the agent and associate it with my Qualys subscription. Shortly after this I would expect to see the new Fedora system appear as a new system in the Qualys portal. Half an hour passed. Nothing.
Looking in /var/log/qualys/qualys-cloud-agent.log I could see the following entries:
2019-01-20 16:15:37.173 [qualys-cloud-agent][1657]:[Error]:Http request failed:HTTP response code said error: The requested URL returned error: 500 Internal Server Error
2019-01-20 16:15:37.173 [qualys-cloud-agent][1657]:[Error]:Http request failed: error code: 500
2019-01-20 16:15:37.173 [qualys-cloud-agent][1657]:[Error]:CAPI request failed:
2019-01-20 16:15:37.173 [qualys-cloud-agent][1657]:[Error]:CAPI event failed
2019-01-20 16:15:37.173 [qualys-cloud-agent][1657]:[Information]:Server busy error: adding network delay: 450 seconds
I had a search on the web, but it bore no fruit.
I fired up wireshark to see if there were any clues in the network traffic.
I noticed that there was a DNS lookup for ‘metadata.google.internal’. ‘What did Google have to do with anything?’ I thought. I found this Qualys Press Release from November 2018 announcing new integration with the Google Cloud Platform.
The response for the above DNS query was as follows:
metadata.google.internal: type A, class IN, addr 92.242.132.24
The address 92.242.132.24 is the IP address returned by Virgin Media’s DNS servers when there is no such DNS record. The idea being that if you’re using a web browser and click on a link to a site that doesn’t exist in DNS, you’ll be ‘helpfully’ presented with Virgin’s advancedsearch2.virginmedia.com page.
My gut feeling was that getting this DNS response might be confusing the agent, leading to the error.
As a test, I edited /etc/resolve.conf on the Fedora machine to point to Google’s DNS server 8.8.8.8.
I then restarted the Qualys Cloud Agent service:
service qualys-cloud-agent restart
and checked the logs again:
2019-01-20 16:22:29.560 [qualys-cloud-agent][1854]:[Information]:Http request completed successfully: 200
2019-01-20 16:22:29.563 [qualys-cloud-agent][1854]:[Warning]:Instance-id found empty, retry Count: 1
2019-01-20 16:22:29.626 [qualys-cloud-agent.provision][1854]:[Debug]:Found agent HostID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
2019-01-20 16:22:29.627 [qualys-cloud-agent][1854]:[Information]:CAPI event successfully completed
2019-01-20 16:22:29.635 [qualys-cloud-agent][1854]:[Information]:Next event: INTERVAL_EVENT_CONFIG, time left: 60 seconds
That looked more promising!
The new Google DNS response in wireshark looked as follows:
Flags: 0x8183 Standard query response, No such name
Checking in the Qualys portal a few moments later…. the Fedora system was listed. Good times!
Crazy how the simple choice of a DNS server can have such unexpected consequences.
After a bit of research, it looks like I can opt out of the ‘advanced error search’ here.