When Checking the Box Results in Two Zero Days and Root (CVE-2019-14257 and CVE-2019-14258)

Finding new bugs and exploiting them can be exciting and fun for a penetration tester. I was ecstatic to find my first two zero-days, and I used them to break a system from no access to root. This was a good day for me – but the story behind the story provides some real lessons enterprises can apply to their security programs.

Taking security seriously and continuously improving

As a penetration tester, we run into networks that have serious issues far too often. These are networks that cause a basic vulnerability scan to be filled with high-, medium-, and low-severity findings. Sadly, because time is a finite resource on an engagement, many clients don’t get the full value that they could out of a network pentest. These sorts of networks make it hard for a pentester to dig deep, because we spend a large amount of time just exploiting and reporting on the high volume caught by a vulnerability scan, even though many of the items will not likely lead to compromise during a penetration test.

On this most auspicious day of penetration testing, I had a client who takes security seriously. They had worked with Coalfire for a few years doing their PCI testing, and it was my second year in a row conducting the testing on their behalf. What set this situation apart was that the client implemented the changes from the previous year’s pentest report all the way down to the low-severity SSL findings. This year, the environment looked fairly clean. I had the time to go through and validate all the low-impact findings as usual, and then started looking at what was most interesting next.

Increased value

Because this client’s network was fairly clean, I was able to dig deeper and find an issue that was new and would not be found by any products or typical scanners. Scanners are great at finding known vulnerabilities; however, penetration tests are about showing the impact of vulnerabilities and identifying things the scanners can’t or don’t identify. 

During a compliance engagement, the tester is focusing on getting at a target. For PCI-DSS tests, the objective is to get to the credit card data. If the network is clean, patched, and well configured, it can be difficult to get that initial foothold that sends a tester down the path that can lead to a demonstration of risk to the client.

How they were found

A standard part of penetration tests is to look at all the web services. One of these had a strange error: “HTTP ERROR: 405 Problem Accessing /. Reason: HTTP method GET is not supported by this URL.”

This was a get request, so the error stood out. Examining it with Burp, I tried to make a post request. This time the error changed to “Failed to parse XML-RPC request: Premature end of file.”

This was starting to sound like something a mentor here at Coalfire told me: “Sometimes an API will tell you what to do so you can exploit it.” With this in mind, I looked up an example of an XML RPC request. I put that in the body of the POST request, and another error was identified: “No such handler.”

Since I hadn’t looked at the rest of the server to figure out what this service was for, I let Burp Suite Pro run its scanner on the request. I saw that this was a Zenoss server, and there were other ports open.

Looking back at Burp, there was an XML XXE finding. I was staring at the output from /etc/passwd. At this point, I have a Linux system and the ability to read files. At the very least I’m happy to be making progress. Manually looking at the XXE, I viewed some other files and found that I could also list directories.

click to enlarge image

Once I could list directories through the web service, I was no longer blindly enumerating the OS. This made life much easier, and now I was convinced that I could get a shell on the system. Motivated by this, I kept up the enumeration. Looking at the directory listing of /home I only saw a couple of users. Checking each of the directories revealed that I had access as the Zenoss user. Another juicy tidbit showed up as well. I could see the /home/zenoss/.ssh/ directory, and there was an authorized_keys file. This is the file that allows a user to log in to a system over SSH using a private key. A separate configuration issue was present where, quite fortunately for me, the id_rsa.pub public key was also in the authorized_keys file. I read out the private key and logged in as the Zenoss user over SSH. This was a classic case of “that one guy built that one server and he’s been gone for . . . ”.

This was an anomaly on the network, so it provided more value. I was able to show them something that the usual tools couldn’t.

There was another user that I wanted to get into the home directory and see if there was anything that I could find that would get me into the CDE. This was the network monitoring system after all; who knows what kind of gems a pentester can find on a succulent target like this?

I wanted root. I started with the classic search for SUID and SGID binaries. The usual binaries showed up, but a couple of others were new to me. The one that led to compromise was pyraw. This was apparently part of the Zenoss distribution due to its location. I found that it would execute arbitrary python, but it dropped the SUID privileges before it executed the code I provided. I tried running strings on the binary, and it spit out some interesting python code.

I’m not an experienced reverse engineer, and Ghidra had just come out, so I figured I would give it a shot. I threw the pyraw executable in, and it spit out some decent looking C source. I did a quick search for the python code I had seen when running strings on the binary and started to get my bearings.

I traced the execution back, and it looked like the python code I had seen was run with elevated permissions, but the permissions were dropped after that, and then the user-provided code was run. I consulted my co-worker and generally brilliant researcher Victor Teissler (@VTeissler on Twitter). He confirmed that I was seeing what I thought I was seeing, and we got to work figuring out how to privesc to root.

Victor suggested that we could hijack the site.py library to execute arbitrary code before the privileges were dropped. We did some testing, and about 15 minutes after identifying the vulnerability, we had a working proof of concept that resulted in root permissions. First, we had to create a new directory structure to hold the modified site.py, but the exploit script was short and easy to understand.

That is how I ended up with two CVEs when doing a compliance penetration test. Naturally, I continued testing, finishing out the rest of the network. While the CVEs were great to get for me personally, I was happy that I could provide a solid pentest to the client and help them to shine a light in the dark corners of their environment. I was also pleased to work on a client project where the customer applied solid best practices, enabling the penetration test to provide them with full value.

The CVEs have been released and exploits have been published. We provided Zenoss with more than 90 days for responsible disclosure to fix the vulnerabilities and release a patch. When this post was published, no patch had been released, however, the manual remediation guidance provided by Zenoss is included at the end of this post. I would like to thank the Zenoss security point of contact for being a cordial and pleasant person to work with during the responsible disclosure process.

Responsible Disclosure Timeline

The initial discovery of the vulnerability occurred during the week of 4/15/2019. The client authorized Coalfire to pursue responsible disclosure on 4/22/2019. The first response from Zenoss, Inc., was on 4/24/2019, and Coalfire provided exploitation details and proof of concept code.

Zenoss asked that Coalfire include remediation guidance with this post. The following guidance has been provided by Zenoss, Inc.:

ZENOSS REMEDIATION GUIDANCE: 

You can reach Zenoss for the latest status on these CVEs using our support page or post in the Zenoss Community forums.

ZenJMX: XML External Entity vulnerability in zenjmx service (CVE-2019-14258)

Starting with version 5.0.0, the Zenoss Service Dynamics product was rearchitected to move the daemon processes running on the host to individual docker containers called services.

Consequently port 9988 is no longer publicly exposed but still available to other docker containers within the ZSD product.

The vulnerability is currently in our backlog and next year is the earliest time a fixed version could be released. Users can monitor the ZenPack documentation for future updates.

In the meantime, the following workarounds can be used to mitigate the exposed vulnerability in all versions.

1. Manually patching the "Java 2 Platform Standard Edition - J2E" ZenPack a.k.a ZenJMX ZenPack

    

   Manually replace "xmlrpc-common-3.0.jar" and "xmlrpc-server-3.0.jar" found at /opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.12.1-py2.7.egg/ZenPacks/zenoss/ZenJMX/lib/ with the latest (apache-xmlrpc-3.1.3) jars, found at http://archive.apache.org/dist/ws/xmlrpc/

    

   Note: The file location of ZenPacks (/opt/zenoss/ZenPacks) may vary depending on ZSD version.

    

   Note 2: The file location contains the version number of the ZenJMX ZenPack. In the above steps we use the latest version "ZenPacks.zenoss.ZenJMX-3.12.1-py2.7.egg"

2. Stopping/disabling the ZenJMX service / daemon

    

   If you are not using the JMX monitoring capabilities provided by this ZenPack then you can safely stop/disable the daemon/service which will result in no longer listening on port 9988
   In 4.2.5 remove the zenjmx daemon from $ZENHOME/etc/daemons.txt. Stop/kill the zenjmx daemon process.

    

   In 5.0.0+, use serviced to edit the zenjmx service "Launch" setting to "manual" e.g. "serviced service edit zenjmx" and search the file for "Launch". Then stop the zenjmx service in the Control Center UI.

3. Remove the ZenJMX service / daemon

    

   In 4.2.5 you can remove the ZenJMX Zenpack in the ZenPacks tab under Settings

    

   In 5.0.0+, follow the current documentation on ZenPack removal. https://help.zenoss.com/zsd/RM/administering-resource-manager/extending-resource-manager-with-zenpacks/removing-a-zenpack

    

   Note: Upgrading the version of ZSD may bring back the ZenJMX ZenPack. If that occurs, repeat the un-install.

Privilege escalation vulnerability in /opt/zenoss/bin/pyraw (CVE-2019-14257)

The vulnerability is currently being worked on by our engineering team. We expect to release a fix late this year or early next year.

Please note that Zenoss 4.2.5 reached End of Maintenance on 2018-06-30 and the fix will only be included in versions after our newest release (6.4+) and then backported to versions still under maintenance (6.2, 6.3)

https://support.zenoss.com/hc/en-us/articles/202991725-End-of-Maintenance-Dates

Starting with version 5.0.0, the Zenoss Service Dynamics product was rearchitected to move the daemon processes running on the host to individual docker containers called services. Consequently, the privilege escalation will occur within a running docker container. This provides only minimal mitigation as the attacker can then proceed to attack other containers or attempt to escape the container onto the host.

In the meantime, the following work around can be used to mitigate the exposed vulnerability in all versions.

1. Removal of pyraw

    

   In 4.2.5, pyraw is used by the discovery feature. If you are not using the discovery feature then you can safely remove the binary pyraw.

    

   In 5.0.0+, pyraw is no longer used and can be safely removed.

2. Exposing pyraw temporarily

    

   When not needing the discovery feature, render the binary unusable
     chmod 000 /opt/zenoss/bin/pyraw

    

   If you need to use the discovery feature, Zenoss recommends you add back the needed permissions and then once done, re-render the file unusable   chmod 04750 /opt/zenoss/bin/pyraw

    

   Note: In version 5.0.0+ a change to a container can only be made permanent by committing the container snapshot with the change

    

   Note 2: Upgrading the version of ZSD may bring back pyraw and/or the file permissions. If that occurs, repeat the removal / render unusable.

    

   Note 3: The file location of pyraw (/opt/zenoss/bin/pyraw) may vary depending on ZSD version.