SeaCat and OpenSSL Heartbleed Bug

Update (5th September 2016)

After almost two and a half year we hope that the Heartbleed remains in the past. It is not true, unfortunately. Based on Censys weekly Heartbleed scan report, ^[1] we found that more than 30,000 out of the top one million domains ^[2] are still vulnerable. ^[3][4] Censys tested one million top sites, but there are more than one billion websites on the Internet. ^[5] If we assume that at around 43% of websites offer HTTPS connections ^[6], and the percentage of the vulnerability will be the same for the remaining 999 millions of websites (probably not and the results will be worse), we have 15 millions of vulnerable servers. This is a big number for hackers and cyber attackers to inflict damage to those server operators. We are surprised that the number is that high.

Now we have proof that a security vulnerability remains with us for a long time, maybe almost forever even when there exist patches and fixes. The Internet is a battlefield among the good, the bad, and the ugly. Who has better attacking or defending technology wins.

We hope that the number of vulnerable servers will be almost zero in the next five years. Are we optimistic? What do you think?

Update (10th April 2014): Original content of this blog entry stated that one of our SeaCat servers detected a "Heartbleed" bug attack prior its actual disclosure. EFF ^[7] correctly pointed out that there are other tools that can produce the same pattern in the SeaCat server log. ^[8] I don't have any hard data evidence to support or reject this statement. Since there is a risk that our finding is a false positive, I have modified this entry to a neutral tone and have removed any conclusions. There are real honeypots on the Internet that should provide final evidence when Heartbleed is broadly exploited for the first time.

Please also see another update attached to the end of this post. There is a lot of attention surrounding this entry.

Original article (7th April 2014)

Today, on the 7th April 2014, a serious vulnerability in the popular OpenSSL has been discovered and published. This bug is especially nasty since it can disclose important secret information in an undetectable way.

SeaCat server is using OpenSSL and therefore it is exposed to this vulnerability. However, thanks to very selective SSL handshake implementation, it seems that SeaCat server rejects to respond to a TLS heartbeat request until authorisation is completed making it resistant to this threat.

Every fresh distribution of the SeaCat server uses a fixed version of OpenSSL, mitigating this issue completely.

The SeaCat client for iOS and Java/Android is not affected since it uses other SSL implementation than OpenSSL (platform specific) and it is running in client mode.

How does Heartbleed bug work?

Photo credit: https://imgs.xkcd.com/comics/heartbleed_explanation.png

Test of the SeaCat server that uses a compromised OpenSSL version

We have tested this vulnerability using a simple detection tool, hb-test.py ^[9]

This is an output:

$ python hb-test.py 192.241.86.55 -p 443
Connecting...
Sending Client Hello...
Waiting for Server Hello...
 ... received message: type = 22, ver = 0301, length = 58
 ... received message: type = 22, ver = 0301, length = 2914
 ... received message: type = 22, ver = 0301, length = 14

Unexpected EOF receiving record header - server closed connection
Server closed connection without sending Server Hello.

The SeaCat server refuses to communicate and closes the connection after a while. Moreover, there is a warning in the log file (see below). No internal information has been exposed since no heartbeat packet has been returned.

You should upgrade your OpenSSL library anyway!

Log entries

Since the SeaCat server creates a log entry for such an attack attempt, it is actually representing a kind of honeypot. The same entry is also created for any other non-standard SSL handshake that is not necessarily an attack (e.g. mass scan tool).

Here are log entries observed by two of our SeaCat servers. These entries started on the 23rd March and culminated on the 7th April 2014 (the day OpenSSL vulnerability was announced publicly). IP addresses of these servers are quite private (only DNS entries exist) and service is running on port TCP/443 (standard HTTPS), so these attempts must be the result of broad automated scans.

Server 1

2014:03:23 07:24:48     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:03:24 11:42:42     691  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:03:24 12:43:26     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:03:25 03:23:03     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:03:28 09:22:30     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:03:31 02:18:55     691  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:03:31 08:06:13     682  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:03:31 21:19:07     691  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:04:04 20:46:17     687  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:04:07 08:20:36     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:04:07 11:25:46     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:04:07 11:25:47     691  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:04:07 11:25:49     682  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:04:07 11:25:53     691  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)
2014:04:07 11:25:55     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:04:07 11:25:58     687  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:04:08 13:20:36     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:04:08 21:25:54     687  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib
2014:04:08 21:27:11     691  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

Server 2

Luckily, This server has debug level logging enabled.

2014:03:24 09:41:52     690 DEBUG: New client connection accepted from 54.186.135.68 49769
2014:03:24 09:42:12     690  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:03:24 10:12:42     696 DEBUG: New client connection accepted from 220.181.159.76 48950
2014:03:24 10:12:42     696  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)

2014:03:25 06:36:56     699 DEBUG: New client connection accepted from 54.193.75.203 36633
2014:03:25 06:37:17     699  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:03:27 08:33:30     696 DEBUG: New client connection accepted from 5.61.43.116 43177
2014:03:27 08:34:13     696  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:03:31 05:43:09     690 DEBUG: New client connection accepted from 173.45.70.84 44362
2014:03:31 05:43:09     690  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)

2014:03:31 10:01:10     699 DEBUG: New client connection accepted from 54.72.158.29 44537
2014:03:31 10:01:30     699  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:03:31 21:19:07     685 DEBUG: New client connection accepted from 113.64.221.42 1919
2014:03:31 21:19:07     685  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)

2014:04:04 20:45:20     685 DEBUG: New client connection accepted from 83.222.250.151 58947
2014:04:04 20:45:20     685  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)

2014:04:06 04:19:09     699 DEBUG: New client connection accepted from 62.210.125.141 33877
2014:04:06 04:19:09     699  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:04:07 09:32:16     696 DEBUG: New client connection accepted from 54.197.30.237 60999
2014:04:07 09:32:36     696  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:04:07 11:32:32     696 DEBUG: New client connection accepted from 80.82.70.118 55409
2014:04:07 11:32:32     696  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:04:07 11:32:34     696 DEBUG: New client connection accepted from 80.82.70.118 60641
2014:04:07 11:32:34     696  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)

2014:04:07 11:32:42     690 DEBUG: New client connection accepted from 80.82.70.106 49246
2014:04:07 11:32:42     690  WARN: Unexpected SSL error during handshake (1): error:00000001:lib(0):func(0):reason(1)

2014:04:07 11:32:42     685 DEBUG: New client connection accepted from 80.82.70.106 49322
2014:04:07 11:32:44     685  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:04:07 11:32:45     690 DEBUG: New client connection accepted from 80.82.70.106 50842
2014:04:07 11:32:47     690  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:04:08 13:19:11     699 DEBUG: New client connection accepted from 209.126.230.70 59259
2014:04:08 13:19:23     699  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

2014:04:08 18:23:44     690 DEBUG: New client connection accepted from 183.60.244.46 43715
2014:04:08 18:23:49     690  WARN: Unexpected SSL error during handshake (5): error:00000005:lib(0):func(0):DH lib

Update (10th April 2014)

There is quite a large number of people reading this post and asking similar questions.
Here is a small FAQ:

What are these servers?

These servers are test/demo instances of our product. It is one virtual server with multiple public IPs that are exposed to the Internet (it is a part of a test).

It is a long-running stability test of a new SeaCat product: people from our Czech branch have a dedicated mobile application on their phones that communicates with these servers via the Internet.
From time to time, we also observe random scans (that is not surprising since it is running on TCP/443.)

This mobile application is not published into any store, and it is quite limited to quite a specific group of people. The current configuration was enabled during January 2014, and the last significant change was implemented in February 2014.

Since that time, it has run quite smoothly. I actually noticed these log entries even before the 'Heartbleed bug announcement' and they even made it into our development backlog, but not much priority has been assigned to them since the server is stable.

Can you describe the architecture of the server software?

It is built using libev, meaning that it is asynchronous in terms of IO.
This implicates that OpenSSL is not reading data directly from a network socket (and not writing data to a socket), but there is a layer between the actual socket and OpenSSL.

The OpenSSL handshake is also coordinated by our code. This is not particularly extraordinary design - a lot of even somewhat advanced servers are designed this way.

However, this gives you a possibility to 1) log activities that deviate from a 'normal' connection, and 2) insert additional checks into a standard OpenSSL handshake.

Since our product encompasses client implementation, we narrowed down a way for SSL to establish a secure session, and excluded techniques that we don't need or want to support.

I guess, in this way, it actually complains about Heartbleed attack attempts - it is not following the expected path that our clients do. This also applies to any other non-standard SSL handshake; I'm not able to retrospectively see a difference.

What does it mean?

• I don't want to state that the SeaCat server is/was resistant to Heartbleed attacks; it just rejects instances we have tested for so far and emits a log message.
• I won't even say that every Heartbleed attack was logged; I can imagine that some managed to fit inside SeaCat handshake expectations, and no log entry was emitted.
• I can say that we started to observe these log entries on the 24th of March, and the root cause was not clear to us; yesterday I was able to produce the same log entry by using a Heartbleed attack script. My explanation is that someone was testing some kind of SSL handshake type of attack broadly on the Internet - maybe Heartbleed or maybe it was just a mass scan. We haven't observed this during January, February or first half of March.
• I admit this is quite a lucky coincidence. I would love to see some reports from real honeypots.

Can I test that personally?

Yes, you can test that on your own: SeaCat is available as a trial for Mac, and you can virtually recreate the whole story yourself.

Reference

1. https://censys.io/data/443-https-heartbleed-alexa_top1mil
2. http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
3. https://scans.io/zsearch/npnb1jr189rztggf-443-https-heartbleed-alexa_top1mil-20160904T130001-zgrab-log.log.lz4
4. https://filippo.io/Heartbleed/faq.html
5. http://www.internetlivestats.com/total-number-of-websites
6. https://en.wikipedia.org/wiki/HTTPS
7. https://www.eff.org/
8. http://blog.erratasec.com/2014/04/no-we-werent-scanning-for-hearbleed.html#.U0Xq4OaSz-l
9. https://gist.githubusercontent.com/takeshixx/10107280/raw/8052d8479ad0c6150464748d639b0f5e877e8c37/hb-test.py

About the Author

Ales Teska

TeskaLabs’ founder and CEO, Ales Teska, is a driven innovator who proactively builds things and comes up with solutions to solve practical IT problems.