For those of you who aren’t familiar with the concept of a network tarpit it is a fairly simple concept. Wikipedia defines it like this:
A tarpit is a service on a computer system (usually a server) that purposely delays incoming connections. The technique was developed as a defense against a computer worm, and the idea is that network abuses such as spamming or broad scanning are less effective, and therefore less attractive, if they take too long. The concept is analogous with a tar pit, in which animals can get bogged down and slowly sink under the surface, like in a swamp.
https://en.wikipedia.org/wiki/Tarpit_(networking)
If you run any sort of service on the internet then you know as soon as your server has a public IP address and open ports, there are scanners and bots trying to get in constantly. If you take decent steps towards security then it is little more than an annoyance, but annoying all the less. One day when I had some extra time on my hands I started researching ways to mess with the bots trying to scan/attack my site.
It turns out that this problem has been solved multiple times in multiple ways. One of the most popular tools for tarpitting ssh connections is endlessh. The way it works is actually pretty simple. The SSH RFC states that when an SSH connection is established, both sides MUST send an identification string. Further down the spec is the line that allows this behavior:
The server MAY send other lines of data before sending the version string. Each line SHOULD be terminated by a Carriage Return and Line Feed. Such lines MUST NOT begin with "SSH-", and SHOULD be encoded in ISO-10646 UTF-8 [RFC3629] (language is not specified). Clients MUST be able to process such lines. Such lines MAY be silently ignored, or MAY be displayed to the client user. If they are displayed, control character filtering, as discussed in [SSH-ARCH], SHOULD be used. The primary use of this feature is to allow TCP- wrappers to display an error message before disconnecting.SSH RFC
Essentially this means that their is no limit to the amount of data that a server can send back to the client and the client must be able to wait and process all of this data. Now let’s see it in action.
git clone https://github.com/skeeto/endlessh.git
cd endlessh
make
./endlessh &By default this fake server listens on port 2222. I have a port forward set up that forwards all ssh traffic from port 22 to 2222. Now try to connect via ssh:
ssh -vvv localhost -p 2222If you wait a few seconds you will see the server send back the version string and then start sending a random banner:
$:/tmp/endlessh$ 2024-06-24T13:05:59.488Z Port 2222
2024-06-24T13:05:59.488Z Delay 10000
2024-06-24T13:05:59.488Z MaxLineLength 32
2024-06-24T13:05:59.488Z MaxClients 4096
2024-06-24T13:05:59.488Z BindFamily IPv4 Mapped IPv6
2024-06-24T13:05:59.488Z socket() = 3
2024-06-24T13:05:59.488Z setsockopt(3, SO_REUSEADDR, true) = 0
2024-06-24T13:05:59.488Z setsockopt(3, IPV6_V6ONLY, true) = 0
2024-06-24T13:05:59.488Z bind(3, port=2222) = 0
2024-06-24T13:05:59.488Z listen(3) = 0
2024-06-24T13:05:59.488Z poll(1, -1)
ssh -vvv localhost -p 2222
OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/mikeconrad/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mikeconrad/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mikeconrad/.ssh/known_hosts2'
debug2: resolving "localhost" port 2222
debug3: resolve_host: lookup localhost:2222
debug3: ssh_connect_direct: entering
debug1: Connecting to localhost [::1] port 2222.
debug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x10
debug1: Connection established.
2024-06-24T13:06:08.635Z = 1
2024-06-24T13:06:08.635Z accept() = 4
2024-06-24T13:06:08.635Z setsockopt(4, SO_RCVBUF, 1) = 0
2024-06-24T13:06:08.635Z ACCEPT host=::1 port=43696 fd=4 n=1/4096
2024-06-24T13:06:08.635Z poll(1, 10000)
debug1: identity file /home/mikeconrad/.ssh/id_rsa type 0
debug1: identity file /home/mikeconrad/.ssh/id_rsa-cert type 4
debug1: identity file /home/mikeconrad/.ssh/id_ecdsa type -1
debug1: identity file /home/mikeconrad/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/mikeconrad/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/mikeconrad/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/mikeconrad/.ssh/id_ed25519 type -1
debug1: identity file /home/mikeconrad/.ssh/id_ed25519-cert type -1
debug1: identity file /home/mikeconrad/.ssh/id_ed25519_sk type -1
debug1: identity file /home/mikeconrad/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/mikeconrad/.ssh/id_xmss type -1
debug1: identity file /home/mikeconrad/.ssh/id_xmss-cert type -1
debug1: identity file /home/mikeconrad/.ssh/id_dsa type -1
debug1: identity file /home/mikeconrad/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
2024-06-24T13:06:18.684Z = 0
2024-06-24T13:06:18.684Z write(4) = 3
2024-06-24T13:06:18.684Z poll(1, 10000)
debug1: kex_exchange_identification: banner line 0: V
2024-06-24T13:06:28.734Z = 0
2024-06-24T13:06:28.734Z write(4) = 25
2024-06-24T13:06:28.734Z poll(1, 10000)
debug1: kex_exchange_identification: banner line 1: 2I=ED}PZ,z T_Y|Yc]$b{R]
This is a great way to give back to those bots and script kiddies. In my research into other methods I also stumbled across this brilliant program fakessh. While fakessh isn’t technically a tarpit, it’s more of a honeypot but very interesting nonetheless. It creates a fake SSH server and logs the ip address, connection string and any commands executed by the attacker. Essentially it allows any username/password combination to connect and gives them a fake shell prompt. There is no actual access to any file system and all of their commands basically return gibberish.
Here are some logs from an actual server of mine running fakessh
2024/06/24 06:51:20 [conn] ip=183.81.169.238:40430
2024/06/24 06:51:22 [auth] ip=183.81.169.238:40430 version="SSH-2.0-Go" user="root" password="0"
2024/06/24 06:51:23 [conn] ip=183.81.169.238:40444
2024/06/24 06:51:25 [auth] ip=183.81.169.238:40444 version="SSH-2.0-Go" user="root" password="eve"
2024/06/24 06:51:26 [conn] ip=183.81.169.238:48408
2024/06/24 06:51:27 [auth] ip=183.81.169.238:48408 version="SSH-2.0-Go" user="root" password="root"
2024/06/24 06:51:28 [conn] ip=183.81.169.238:48434
2024/06/24 06:51:30 [auth] ip=183.81.169.238:48434 version="SSH-2.0-Go" user="root" password="1"
2024/06/24 06:51:30 [conn] ip=183.81.169.238:48448
2024/06/24 06:51:32 [auth] ip=183.81.169.238:48448 version="SSH-2.0-Go" user="root" password="123"
2024/06/24 06:51:32 [conn] ip=183.81.169.238:48476
2024/06/24 06:51:35 [auth] ip=183.81.169.238:48476 version="SSH-2.0-Go" user="root" password="admin"
2024/06/24 06:51:35 [conn] ip=183.81.169.238:39250
2024/06/24 06:51:37 [auth] ip=183.81.169.238:39250 version="SSH-2.0-Go" user="root" password="123456"
2024/06/24 06:51:38 [conn] ip=183.81.169.238:39276
2024/06/24 06:51:40 [auth] ip=183.81.169.238:39276 version="SSH-2.0-Go" user="root" password="123123"
2024/06/24 06:51:40 [conn] ip=183.81.169.238:39294
2024/06/24 06:51:42 [auth] ip=183.81.169.238:39294 version="SSH-2.0-Go" user="root" password="test"
2024/06/24 06:51:43 [conn] ip=183.81.169.238:39316
2024/06/24 06:51:45 [auth] ip=183.81.169.238:39316 version="SSH-2.0-Go" user="root" password="123456789"
2024/06/24 06:51:45 [conn] ip=183.81.169.238:35108
2024/06/24 06:51:47 [auth] ip=183.81.169.238:35108 version="SSH-2.0-Go" user="root" password="12345"
2024/06/24 06:51:48 [conn] ip=183.81.169.238:35114
2024/06/24 06:51:50 [auth] ip=183.81.169.238:35114 version="SSH-2.0-Go" user="root" password="password"
2024/06/24 06:51:50 [conn] ip=183.81.169.238:35130
2024/06/24 06:51:52 [auth] ip=183.81.169.238:35130 version="SSH-2.0-Go" user="root" password="12345678"
2024/06/24 06:51:52 [conn] ip=183.81.169.238:35146
2024/06/24 06:51:54 [auth] ip=183.81.169.238:35146 version="SSH-2.0-Go" user="root" password="111111"
2024/06/24 06:51:55 [conn] ip=183.81.169.238:58490
2024/06/24 06:51:57 [auth] ip=183.81.169.238:58490 version="SSH-2.0-Go" user="root" password="1234567890"
2024/06/24 06:51:57 [conn] ip=183.81.169.238:58528
2024/06/24 06:51:59 [auth] ip=183.81.169.238:58528 version="SSH-2.0-Go" user="root" password="1234"
2024/06/24 06:52:00 [conn] ip=183.81.169.238:58572
2024/06/24 06:52:02 [auth] ip=183.81.169.238:58572 version="SSH-2.0-Go" user="root" password="password123"
2024/06/24 06:52:02 [conn] ip=183.81.169.238:58588
2024/06/24 06:52:04 [auth] ip=183.81.169.238:58588 version="SSH-2.0-Go" user="root" password="ubuntu"
2024/06/24 06:52:05 [conn] ip=183.81.169.238:37198
2024/06/24 06:52:07 [auth] ip=183.81.169.238:37198 version="SSH-2.0-Go" user="Antminer" password="root"
2024/06/24 06:52:07 [conn] ip=183.81.169.238:37214
2024/06/24 06:52:09 [auth] ip=183.81.169.238:37214 version="SSH-2.0-Go" user="Antminer" password="admin"
2024/06/24 06:52:10 [conn] ip=183.81.169.238:37238
2024/06/24 06:52:11 [auth] ip=183.81.169.238:37238 version="SSH-2.0-Go" user="root" password="innot1t2"
2024/06/24 06:52:12 [conn] ip=183.81.169.238:37258
2024/06/24 06:52:14 [auth] ip=183.81.169.238:37258 version="SSH-2.0-Go" user="root" password="t1t2t3a5"
2024/06/24 06:52:14 [conn] ip=183.81.169.238:55658
2024/06/24 06:52:16 [auth] ip=183.81.169.238:55658 version="SSH-2.0-Go" user="root" password="blacksheepwall"
2024/06/24 06:52:17 [conn] ip=183.81.169.238:55670
2024/06/24 06:52:19 [auth] ip=183.81.169.238:55670 version="SSH-2.0-Go" user="root" password="envision"
2024/06/24 06:52:19 [conn] ip=183.81.169.238:55708
2024/06/24 06:52:21 [auth] ip=183.81.169.238:55708 version="SSH-2.0-Go" user="root" password="bwcon"
2024/06/24 06:52:22 [conn] ip=183.81.169.238:55776
2024/06/24 06:52:23 [auth] ip=183.81.169.238:55776 version="SSH-2.0-Go" user="admin" password="root"
2024/06/24 06:52:24 [conn] ip=183.81.169.238:46646
2024/06/24 06:52:26 [auth] ip=183.81.169.238:46646 version="SSH-2.0-Go" user="baikal" password="baikal"
2024/06/24 06:52:26 [conn] ip=180.101.88.197:44620
2024/06/24 06:52:27 [conn] ip=180.101.88.197:44620 err="ssh: disconnect, reason 11: "
2024/06/24 06:53:35 [conn] ip=218.92.0.76:50610
2024/06/24 06:53:36 [conn] ip=218.92.0.76:50610 err="ssh: disconnect, reason 11: "
2024/06/24 07:02:28 [conn] ip=218.92.0.27:64676
2024/06/24 07:02:30 [conn] ip=218.92.0.27:64676 err="ssh: disconnect, reason 11: "
2024/06/24 07:10:05 [conn] ip=218.92.0.76:57601
2024/06/24 07:10:07 [conn] ip=218.92.0.76:57601 err="ssh: disconnect, reason 11: "
2024/06/24 07:14:05 [conn] ip=193.201.9.156:63056
2024/06/24 07:14:05 [auth] ip=193.201.9.156:63056 version="SSH-2.0-Go" user="ubnt" password="ubnt"
2024/06/24 07:14:05 [conn] ip=193.201.9.156:63056 err="read tcp 10.10.10.107:2222->193.201.9.156:63056: read: connection reset by peer"
2024/06/24 07:24:53 [conn] ip=218.92.0.31:25485
2024/06/24 07:24:54 [conn] ip=218.92.0.31:25485 err="ssh: disconnect, reason 11: "
2024/06/24 07:24:54 [conn] ip=218.92.0.112:39270
2024/06/24 07:24:56 [conn] ip=218.92.0.112:39270 err="ssh: disconnect, reason 11: "
2024/06/24 07:26:42 [conn] ip=218.92.0.34:59993
2024/06/24 07:35:46 [conn] ip=218.92.0.34:59993 err="read tcp 10.10.10.107:2222->218.92.0.34:59993: read: connection reset by peer"
2024/06/24 07:41:28 [conn] ip=218.92.0.107:62285
2024/06/24 07:41:31 [conn] ip=218.92.0.107:62285 err="ssh: disconnect, reason 11: "
2024/06/24 07:43:27 [conn] ip=218.92.0.29:34556
2024/06/24 07:43:28 [conn] ip=218.92.0.29:34556 err="ssh: disconnect, reason 11: "
2024/06/24 07:44:15 [conn] ip=218.92.0.118:37047
2024/06/24 07:44:22 [conn] ip=218.92.0.118:37047 err="ssh: disconnect, reason 11: "
2024/06/24 07:56:10 [conn] ip=157.245.98.245:6116
2024/06/24 07:56:11 [conn] ip=157.245.98.245:6116 err="ssh: unexpected message type 20 (expected 21)"
2024/06/24 07:57:57 [conn] ip=218.92.0.112:28326
2024/06/24 07:57:58 [conn] ip=218.92.0.112:28326 err="ssh: disconnect, reason 11: "
2024/06/24 08:00:01 [conn] ip=218.92.0.24:24948
2024/06/24 08:00:02 [conn] ip=218.92.0.24:24948 err="ssh: disconnect, reason 11: "
2024/06/24 08:06:19 [conn] ip=193.201.9.156:46865
2024/06/24 08:06:20 [auth] ip=193.201.9.156:46865 version="SSH-2.0-Go" user="root" password="xc3511"
2024/06/24 08:06:20 [conn] ip=193.201.9.156:46865 err="read tcp 10.10.10.107:2222->193.201.9.156:46865: read: connection reset by peer"
2024/06/24 08:14:26 [conn] ip=180.101.88.197:48347
2024/06/24 08:14:28 [conn] ip=180.101.88.197:48347 err="ssh: disconnect, reason 11: "
2024/06/24 08:16:28 [conn] ip=218.92.0.56:18064
2024/06/24 08:16:32 [conn] ip=218.92.0.56:18064 err="ssh: disconnect, reason 11: "
2024/06/24 08:30:55 [conn] ip=180.101.88.196:40495
2024/06/24 08:30:57 [conn] ip=180.101.88.196:40495 err="ssh: disconnect, reason 11: "
2024/06/24 08:32:20 [conn] ip=85.209.11.227:15493
2024/06/24 08:32:21 [auth] ip=85.209.11.227:15493 version="SSH-2.0-Go" user="telecomadmin" password="admintelecom"
2024/06/24 08:32:21 [conn] ip=85.209.11.227:15493 err="read tcp 10.10.10.107:2222->85.209.11.227:15493: read: connection reset by peer"
2024/06/24 08:33:19 [conn] ip=218.92.0.34:59804
2024/06/24 08:33:21 [conn] ip=218.92.0.34:59804 err="ssh: disconnect, reason 11: "
2024/06/24 08:41:00 [conn] ip=218.92.0.27:45567
2024/06/24 08:41:02 [conn] ip=218.92.0.27:45567 err="ssh: disconnect, reason 11: "
2024/06/24 08:47:15 [conn] ip=180.101.88.196:17032
2024/06/24 08:47:16 [conn] ip=180.101.88.196:17032 err="ssh: disconnect, reason 11: "
2024/06/24 08:49:51 [conn] ip=218.92.0.29:26360
2024/06/24 08:49:57 [conn] ip=218.92.0.29:26360 err="ssh: disconnect, reason 11: "
2024/06/24 08:58:27 [conn] ip=193.201.9.156:49525
2024/06/24 08:58:28 [auth] ip=193.201.9.156:49525 version="SSH-2.0-Go" user="admin" password="1234"
2024/06/24 08:58:28 [conn] ip=193.201.9.156:49525 err="read tcp 10.10.10.107:2222->193.201.9.156:49525: read: connection reset by peer"
2024/06/24 08:58:44 [conn] ip=218.92.0.31:11835
2024/06/24 08:58:46 [conn] ip=218.92.0.31:11835 err="ssh: disconnect, reason 11: "
2024/06/24 09:03:38 [conn] ip=218.92.0.107:57758
2024/06/24 09:03:40 [conn] ip=218.92.0.107:57758 err="ssh: disconnect, reason 11: "
2024/06/24 09:07:36 [conn] ip=218.92.0.56:21354
2024/06/24 09:07:39 [conn] ip=218.92.0.56:21354 err="ssh: disconnect, reason 11: "
Those are mostly connections and disconnections. They probably connected, realized it was fake and disconnected. There are a couple that tried to execute some commands though:
:~$ sudo grep head /var/log/fakessh/fakessh.log
2024/06/23 15:48:02 [shell] ip=184.160.233.163:45735 duration=0s bytes=15 head="ls 2>/dev/null\n"
2024/06/24 03:55:11 [shell] ip=14.46.116.243:43656 duration=20s bytes=0 head=""
Fun fact: Cloudflare’s Bot Fight Mode uses a form of tarpitting:
Once enabled, when we detect a bad bot, we will do three things: (1) we’re going to disincentivize the bot maker economically by tarpitting them, including requiring them to solve a computationally intensive challenge that will require more of their bot’s CPU; (2) for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; and (3) we’re going to plant trees to make up for the bot’s carbon cost.
https://blog.cloudflare.com/cleaning-up-bad-bots