We run a lot of servers. Sometimes they break, and you have to involve the vendor. I thought I'd write about a severe security issue with the support cases. This is probably well known for most people dealing with vendors, but I don't think there's been enough discussion about it.

The problem

Let's say you have a server which is showing some problems. Maybe a disk is failing, there's networking errors, or you see memory errors are registered in the logs. And let's say your vendor is HP or Dell (since I have had this experience with both of them). It probably affects other vendors too.

Now, as the good admin you are, you open a support case using one of the myriad tools the vendors want you to use. You send the clear log snippet that shows what's wrong, usually completely unambiguous.

Then you get a friendly reply from a level 1 support tech.

Hello, thank you for you support case. Could you please run (insert tool name - usually including sosreport) and attach the logs to the reply mail. This is so we can debug the issue.

Now depending on your experience as an admin you might think, sure, I'll just run a command and send the resulting file to the vendor. Simple enough.

DON'T

At least first look what you're sending. What's in the tarballs, and the tarballs inside the tarballs, and the next level of tarballs.

Let's see the amount of data they want

# ls
peclogs_SERVICETAG
# find . | wc
  11920   11920 1179245

So, in this case, they want ~12 000 files from your server. Usually you find most of /etc/ packed in there.

# find peclogs_SERVICETAG/os/sosreport-username-date/etc/ | wc
   1031    1031   96074
# find /etc/ | wc
   1874    1874   66205

Oh, they do remove some files (passwd, shadow), and in some files they even try to blank out the passwords. First of all, they have NO business digging round there. Secondly without even trying I found 3 plaintext passwords from configuration files, and these would be sent to the vendor. And that's with well managed configuration files. If you have a few extra comments with previously tested credentials, I'm sure these will not be automatically sanitized. And this is only the actual credentials. The amount of sensitive information you don't want spread is much larger.

Of course you'll also send them your sensitive logs. As if that wouldn't be enough, at least in the EU, if you log user logins, you are probably breaking data privacy laws by sending personal information to a third party. And yes, /var/log/secure is sent to the vendor.

The vendors?

Now, all this, will not only be sent to the vendor, which is completely unacceptable, but it possibly be sent as an unencrypted mail attachment. Also, as a customer, I have no knowledge on how the data is handled on the vendor side. I'm sure that if you dig through the vendor support system, you'll find keys to thousands of kingdoms on the internet. This means that these systems are most likely under constant attack, with some successes. And I'm sure that the data is in the hands of several three-letter agencies.

Are the vendors aware of this? Of course they are. They would have to be complete idiots to not notice all the credentials they are sent. But it's probably just easier for them to do it this way.

By any measure, the vendor's actions are completely inexcusable from an IT security point of view.

A better option?

So that this post wouldn't just become a rant, here is a suggestion how to do it better.

Create a tool that

• only collects logs relevant to your current problem.
• clearly states what data it collects, and how it's relevant to the problem.
  
• has options for to collect different data for different problems (e.g. memory, disk).
• at most collects 10 log files at a time. This amount can still be relatively easily sanitized by the admin.
• stays the hell away from /etc/ except /etc/lsb-release
• may collect some standard command outputs, e.g. dmidecode, uname, dmesg, still clearly stating the reason for all of these.
• may collect some per-problem command outputs, e.g. smartctl. Again, for each ouput state why it's needed.
• is not a binary blob. If I run a vendor tool on our systems, I want to be able to verify what it does.
• is not mandatory to run. It's always possible to send relevant snippets of log files instead. With this tool, it should be trivial to create a list of commands to run for manual data collection.

What should you do?

Disclaimer: I have only tested the first two of these.

• Do not send this info to the vendors.
• Call the vendors out on this. Take it up with the them in your quality meetings.
• Have clauses in your contracts that they are not allowed to request this information.
• If the situation does not improve, tell them that any further Phishing attempts will be reported to the authorities.

For those with more time and energy, it would be a nice experiment to actually send the data, but add some honeypot credentials in a believable place in the package, and see if they ever get used.