## Intro

SadServers is a LeetCode style puzzle for Sysadmins/Site Reliability Engineers/DevOps Engineers or whatever Ops people in IT are called nowadays. The following is a writeup of walking through the challenges given.

## Saint John

A developer created a testing program that is continuously writing to a log file `/var/log/bad.log` and filling up disk. You can check for example with `tail -f /var/log/bad.log.` This program is no longer needed. Find it and terminate it.

So let’s see what is accessing this file with `lsof`:

``````\$ lsof /var/log/bad.log
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
``````

Second column lists the ID of the process what writes to the log.

We’ll kill ist with `kill`:

``````kill 621
``````

As a oneliner:

``````kill \$(lsof -t /var/log/bad.log)
``````

## Santiago

Alice the spy has hidden a secret number combination, find it using these instructions:

1) Find the number of lines where the string Alice occurs in `*.txt` files in the `/home/admin` directory 2) There’s a file where “Alice” appears exactly once. In the line after that ocurrence there’s a number. Write both numbers consecutively as one (no new line or spaces) to the solution file (eg if the first number from 1) is 11 and the second 22, you can do `echo -n 11 > /home/admin/solution; echo 22 >> /home/admin/solution`

A little shell tool magic… `grep -Hc file` gives us file names and match counters. `cut` cuts up files on delimiters, `grep -oEe` allows regex-based extraction of values.

``````\$ grep Alice -Hnc *.txt | \
cut -d : -f 2 | \
python3 -c "import sys; print(sum(int(l) for l in sys.stdin))"
``````

Find the word “Alice” in the target `*.txt` files and count the occurences by file, |
select the counters only from the previous grep output and |
sum up the resulting list of numbers with a short python script.

``````grep Alice -Hnc *.txt | grep -Ee :1\$ | cut -d: -f1
``````

Find and count Alice in all target files, |
filter out the file containing only a single occurence |
and get the filename.

``````\$ grep -A1 Alice \
\$(grep Alice -Hnc *.txt | grep -Ee :1\$ | cut -d: -f1) | \
grep -oEe "[0-9]+"
``````

Find the occurence in the file found earlier again, but this time find the line after the matching line with `-A1`, |
select only numbers from the output.

``````echo \${PART1}\${PART2} > /home/admin/solution
``````

Concat both parts, add a newline at the end and write the result to the given file.

As a oneliner:

``````echo \
\$(grep Alice -Hnc *.txt | cut -d : -f 2 | python3 -c "import sys; print(sum(int(l) for l in sys.stdin))")\$(grep -A1 Alice \$(grep Alice -Hnc *.txt | grep -Ee :1\$ | cut -d: -f1) | grep -oEe "[0-9]+") \
``````

There’s a web server access log file at `/home/admin/access.log`. The file consists of one line per HTTP request, with the requester’s IP address at the beginning of each line.

Find what’s the IP address that has the most requests in this file (there’s no tie; the IP is unique). Write the solution into a file `/home/admin/highestip.txt`. For example, if your solution is “1.2.3.4”, you can do `echo "1.2.3.4" > /home/admin/highestip.txt`

``````# -o Only print matching part, -E extended regex, -e pattern is a regex
#  0-255.0-255.0-255.0-255
grep -oEe \
'[12]?[0-9]{1,2}[.][12]?[0-9]{1,2}[.][12]?[0-9]{1,2}[.][12]?[0-9]{1,2}' \
access.log
``````

This yields a list of IP addresses. Now we need to sort and count (`sort | uniq -c`), sort by the leading count, highest at the bottom (`sort -n`), pick the last one (`tail -n1`), and from the resulting line, pick the second field as it’s the IP address. Write it to the result file `> /home/admin/highestip.txt`.

As a oneliner:

``````grep -oEe '[12]?[0-9]{1,2}[.][12]?[0-9]{1,2}[.][12]?[0-9]{1,2}[.][12]?[0-9]{1,2}' access.log | \
sort | \
uniq -c | \
sort -n | \
tail -n1 | \
awk '{ print \$2 }' \
``````

## Tokyo

There’s a web server serving a file `/var/www/html/index.html` with content “hello sadserver” but when we try to check it locally with an HTTP client like `curl 127.0.0.1:80`, nothing is returned. This scenario is not about the particular web server configuration and you only need to have general knowledge about how web servers work.

`ss` tells us that we’re dealing with Apache here:

``````# ss -panetl | grep 80
LISTEN 0      511                *:80              *:*    users:(("apache2",pid=774,fd=4),("apache2",pid=773,fd=4),("apache2",pid=635,fd=4)) ino:17412 sk:1002 cgroup:/system.slice/apache2.service v6only:0 <->
LISTEN 0      4096               *:8080            *:*    users:(("gotty",pid=551,fd=6)) ino:16863 sk:1003 cgroup:/system.slice/gotty.service v6only:0 <->
``````

As it’s Debian, `apache2ctl -S` tells us that there’s no special config here, it’s pretty much boiler plate:

``````# apache2ctl -S
VirtualHost configuration:
*:80                   ip-172-31-21-14.us-east-2.compute.internal (/etc/apache2/sites-enabled/000-default.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex watchdog-callback: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
``````

So let’s just try it:

``````# curl localhost
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<title>403 Forbidden</title>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
</body></html>
``````

So, what does Apache say?

``````# cat /var/log/apache2/error.log
[Mon Dec 26 19:06:44.506725 2022] [mpm_event:notice] [pid 635:tid 140509137360768] AH00489: Apache/2.4.52 (Ubuntu) configured -- resuming normal operations
[Mon Dec 26 19:06:44.506755 2022] [core:notice] [pid 635:tid 140509137360768] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 26 19:11:36.248299 2022] [core:error] [pid 773:tid 140508974245440] (13)Permission denied: [client ::1:52158] AH00132: file permissions deny server access: /var/www/html/index.html
``````

Ah. File permissions. Check it out…

``````# sudo -u www-data namei -mx /var/www/html/index.html
f: /var/www/html/index.html
Drwxr-xr-x /
drwxr-xr-x var
drwxr-xr-x www
drwxr-xr-x html
-rw------- index.html
``````

Ok, path to the file is fine. The permissions on the file itself seem off though…

``````# ls -lhA /var/www/html/
total 4.0K
-rw------- 1 root root 16 Aug  1 00:40 index.html
``````

Jup, Apache’s user `www-data` is not able to access the file.

So change the file’s permissions:

``````chmod a+r /var/www/html/index.html
``````
``````# curl localhost
``````

`curl 127.0.0.1:80` hangs though. Strange. The earlier commands `ss` and `apache2ctl` reveal that Apache listens on all IP addresses, SELinux et al. aren’t active as it’s Debian. So… IPTables shenanigans?

``````# iptables-save
# Generated by iptables-save v1.8.7 on Mon Dec 26 19:19:54 2022
*filter
:INPUT ACCEPT [1677:121936]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1690:328212]
-A INPUT -p tcp -m tcp --dport 80 -j DROP
COMMIT
# Completed on Mon Dec 26 19:19:54 2022
``````

Indeed. `iptables -F` takes care of that, done.

As a oneliner:

``````chmod a+r /var/www/html/index.html ; iptables -F
``````

## Manhattan

Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don’t need to know details about it (although it may help).

Helpful Postgres information: it’s a service that listens to a port (`:5432`) and writes to disk in a data directory, the location of which is defined in the `data_directory` parameter of the configuration file `/etc/postgresql/14/main/postgresql.conf`. In our case Postgres is managed by systemd as a unit with name postgresql.

Can’t write file? Check if the disk is full… With `df -h`:

``````# df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             224M     0  224M   0% /dev
tmpfs             47M  1.5M   46M   4% /run
/dev/nvme1n1p1   7.7G  1.2G  6.1G  17% /
tmpfs            233M     0  233M   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs            233M     0  233M   0% /sys/fs/cgroup
/dev/nvme1n1p15  124M  278K  124M   1% /boot/efi
/dev/nvme0n1     8.0G  8.0G   28K 100% /opt/pgdata
``````

Let’s look into it…

``````# ls -lhA
total 8.0G
-rw-r--r--  1 root     root       69 May 21  2022 deleteme
-rw-r--r--  1 root     root     7.0G May 21  2022 file1.bk
-rw-r--r--  1 root     root     923M May 21  2022 file2.bk
-rw-r--r--  1 root     root     488K May 21  2022 file3.bk
drwx------ 19 postgres postgres 4.0K May 21  2022 main
``````

Purge files and restart the database:

``````# rm -f *.bk
# systemctl restart postgres
``````

As a oneliner:

``````rm -f /opt/pgdata/*.bk ; systemctl restart postgres
``````

## Capetown

There’s an Nginx web server installed and managed by systemd. Running `curl -I 127.0.0.1:80` returns `curl: (7) Failed to connect to localhost port 80: Connection refused`, fix it so when you curl you get the default Nginx page.

Broken nginx, eh? Let’s ask `SystemD` what’s up with it:

``````# systemctl status nginx
● nginx.service - The NGINX HTTP and reverse proxy server
Active: failed (Result: exit-code) since Mon 2022-12-26 19:32:01 UTC; 1min 21s ago
Process: 570 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=1/FAILURE)
CPU: 30ms

Dec 26 19:32:01 ip-172-31-40-160 nginx[570]: nginx: [emerg] unexpected ";" in /etc/nginx/sites-enabled/default:1
Dec 26 19:32:01 ip-172-31-40-160 nginx[570]: nginx: configuration file /etc/nginx/nginx.conf test failed
Dec 26 19:32:00 ip-172-31-40-160 systemd[1]: Starting The NGINX HTTP and reverse proxy server...
Dec 26 19:32:01 ip-172-31-40-160 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Dec 26 19:32:01 ip-172-31-40-160 systemd[1]: nginx.service: Failed with result 'exit-code'.
Dec 26 19:32:01 ip-172-31-40-160 systemd[1]: Failed to start The NGINX HTTP and reverse proxy server.
``````

Ok, so the config’s invalid - the first line is only a ‘;’. Remove it and restart nginx:

``````\$ sed -i -e 1d /etc/nginx/sites-enabled/default
\$ systemctl restart nginx
``````

Test:

``````# curl localhost
<html>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
``````

Damn it, it’s still not working. Check its log file:

``````# cat /var/log/nginx/error.log
2022/12/26 19:37:18 [alert] 961#961: socketpair() failed while spawning "worker process" (24: Too many open files)
2022/12/26 19:37:18 [emerg] 962#962: eventfd() failed (24: Too many open files)
2022/12/26 19:37:18 [alert] 962#962: socketpair() failed (24: Too many open files)
2022/12/26 19:37:22 [crit] 962#962: *1 open() "/var/www/html/index.nginx-debian.html" failed (24: Too many open files), client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost"
``````

Too many open files… Normally, we would check `/etc/security/limits.conf`, but as it’s `SystemD` managed we have to look at the unit:

``````# systemctl cat nginx
# /etc/systemd/system/nginx.service
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecStop=/bin/kill -s QUIT \$MAINPID
PrivateTmp=true
LimitNOFILE=10

[Install]
WantedBy=multi-user.target
``````

Ah great. Edit the unit and reload.

``````# sed -i -e /LimitNOFILE/d /etc/systemd/system/nginx.service
# systemctl restart nginx
# !curl
curl localhost
<!DOCTYPE html>
<html>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
``````

Done.

As a oneliner:

``````sed -i -e 1d /etc/nginx/sites-enabled/default ; \
sed -i -e /LimitNOFILE/d /etc/systemd/system/nginx.service ; \
systemctl restart nginx
``````

## Salta

There’s a “dockerized” Node.js web application in the `/home/admin/app` directory. Create a Docker container so you get a web app on port :8888 and can curl to it. For the solution to be valid, there should be only one running Docker container.

First we check if it’s even broken:

``````admin@ip-172-31-34-225:~/app\$ curl localhost:8888
these are not the droids you're looking for
``````

Huh, odd - somethings listening on that port. What’s going on here, tell us `ss`!

``````\$ ss -panetl | grep 8888
LISTEN 0      511          0.0.0.0:8888      0.0.0.0:*    ino:11536 sk:2 cgroup:/system.slice/nginx.service <->
LISTEN 0      511             [::]:8888         [::]:*    ino:11537 sk:6 cgroup:/system.slice/nginx.service v6only:1 <->
``````

Ah, Nginx - disable it:

``````\$ sudo systemctl disable --now nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable nginx
Removed /etc/systemd/system/multi-user.target.wants/nginx.service.
``````

Ok, so let’s check out what docker is giving us:

``````# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED        STATUS                    PORTS     NAMES
124a4fb17a1c   app       "docker-entrypoint.s…"   3 months ago   Exited (1) 3 months ago             elated_taussig
# docker logs elated_taussig
throw err;
^

Error: Cannot find module '/usr/src/app/serve.js'
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
at node:internal/main/run_main_module:17:47 {
code: 'MODULE_NOT_FOUND',
requireStack: []
}
``````

So it seems that the container image is broken somehow. We’re given the image’s source in `admin`’s homedirectory.

``````admin@ip-xxx:~/app\$ ls
Dockerfile  package-lock.json  package.json  server.js
# documentation https://nodejs.org/en/docs/guides/nodejs-docker-webapp/

# most recent node (security patches) and alpine (minimal, adds to security, possible libc issues)
FROM node:15.7-alpine

# Create app directory & copy app files
WORKDIR /usr/src/app

# we copy first package.json only, so we take advantage of cached Docker layers
COPY ./package*.json ./

# RUN npm ci --only=production
RUN npm install

# Copy app source
COPY ./* ./

# port used by this app
EXPOSE 8880

# command to run
CMD [ "node", "serve.js" ]
``````

Port should be 8888, and the file name in CMD should be `server.js`

Make the changes and rebuild the container image:

``````admin@ip-xxx:~/app\$ sudo docker build -t app:latest .
Sending build context to Docker daemon  101.9kB
Step 1/7 : FROM node:15.7-alpine
---> 706d12284dd5
Step 2/7 : WORKDIR /usr/src/app
---> Using cache
---> 463b1571f18e
Step 3/7 : COPY ./package*.json ./
---> Using cache
---> acfb467c80ba
Step 4/7 : RUN npm install
---> Using cache
Step 5/7 : COPY ./* ./
---> 59c0ca1ef224
Step 6/7 : EXPOSE 8888
---> Running in 2e5faf8ee253
Removing intermediate container 2e5faf8ee253
---> c3127219be52
Step 7/7 : CMD [ "node", "server.js" ]
---> Running in 5d24a99a1a9e
Removing intermediate container 5d24a99a1a9e
---> 3774cc41c752
Successfully built 3774cc41c752
Successfully tagged app:latest
``````

Start the container and check the results:

``````admin@ip-xxx:~/app\$ sudo docker run -d -p 8888:8888 app:latest
397a69aa6832fb6edc922b733fd55c6df169963f30875308287f2298ab99730e
curl localhost:8888
Hello World!
``````

Done

As a oneliner:

``````sudo systemctl disable --now nginx ; \
cd ~/app ; \
sed -i \
-e 's/8880/8888/g' \
-e 's/serve.js/server.js/g' \
Dockerfile ; \
sudo docker build -t app:latest . ; \
sudo docker run -d -p 8888:8888 app:latest
``````

## Jakarta

Can’t ping google.com. It returns `ping: google.com: Name or service not known.` Expected is being able to resolve the hostname. (Note: currently the VMs can’t ping outside so there’s no automated check for the solution).

Check files connected to DNS resolution (`/etc/hosts`, `/etc/resolv.conf`), check relevant services:

``````ubuntu@ip-172-31-42-233:/\$ cat /etc/hosts
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu@ip-172-31-42-233:/\$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
search us-east-2.compute.internal
ubuntu@ip-172-31-42-233:/\$ systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
Active: active (running) since Mon 2022-12-26 20:12:19 UTC; 1min 27s ago
Docs: man:systemd-resolved.service(8)
man:org.freedesktop.resolve1(5)
https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 434 (systemd-resolve)
Status: "Processing requests..."
Memory: 8.2M
CPU: 115ms
CGroup: /system.slice/systemd-resolved.service
└─434 /lib/systemd/systemd-resolved

Dec 26 20:12:18 ip-172-31-42-233 systemd[1]: Starting Network Name Resolution...
Dec 26 20:12:18 ip-172-31-42-233 systemd-resolved[434]: Positive Trust Anchors:
Dec 26 20:12:18 ip-172-31-42-233 systemd-resolved[434]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d0>
Dec 26 20:12:18 ip-172-31-42-233 systemd-resolved[434]: Negative trust anchors: home.arpa 10.in-addr.arp>
Dec 26 20:12:19 ip-172-31-42-233 systemd-resolved[434]: Using system hostname 'ip-172-31-42-233'.
Dec 26 20:12:19 ip-172-31-42-233 systemd[1]: Started Network Name Resolution.
Dec 26 20:12:22 ip-172-31-42-233 systemd-resolved[434]: ens5: Failed to read DNSSEC negative trust ancho>
``````

It’s all good. Is DNS resolution really broken?

``````ubuntu@ip-172-31-42-233:/\$ ping www.google.com
ping: www.google.com: Name or service not known
``````

Yes, it is.

Does DNS resolution work, or is the network to blame?

``````\$ dig www.google.at

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> www.google.at
;; global options: +cmd
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52104
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:

;; Query time: 116 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Dec 26 20:21:23 UTC 2022
;; MSG SIZE  rcvd: 58
``````

So the SystemD resolved cached the entry and things work as they should on that end.

So… is DNS really used by the system? Check `/etc/nsswitch.conf`:

``````ubuntu@ip-172-31-42-233:/\$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:

passwd:         files systemd
group:          files systemd

hosts:          files
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis
``````

The `hosts:` line should look differently. On my box, it looks like this:

``````hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns
``````

So let’s put that in:

``````ubuntu@ip-172-31-42-233:/\$ sudo vim /etc/nsswitch.conf
sudo: unable to resolve host ip-172-31-42-233: Name or service not known
PING www.google.com (142.251.32.4) 56(84) bytes of data.
^C
2 packets transmitted, 0 received, 100% packet loss, time 1005ms
``````

Jup, works now.

As a oneliner:

``````sed \
-i -e 's/hosts:.*/hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns/' \
/etc/nsswitch.conf
``````

## Bern

There are two Docker containers running, a web application (Wordpress or WP) and a database (MariaDB) as back-end, but if we look at the web page, we see that it cannot connect to the database. `curl -s localhost:80 |tail -4` returns:

`<body id="error-page"> <div class="wp-die-message"><h1>Error establishing a database connection</h1></div></body> </html>`

This is not a Wordpress code issue (the image is :latest with some network utilities added). What you need to know is that WP uses “WORDPRESS_DB_” environment variables to create the MySQL connection string. See the `./html/wp-config.php` WP config file for example (from `/home/admin`).

``````root@ip-172-31-19-232:~# docker ps
CONTAINER ID   IMAGE            COMMAND                  CREATED        STATUS              PORTS                    NAMES
6ffb084b515c   wordpress:sad    "docker-entrypoint.s…"   4 months ago   Up About a minute   0.0.0.0:80->80/tcp       wordpress
root@ip-172-31-19-232:~# docker inspect wordpress
[
{
"Created": "2022-08-04T03:22:49.885388997Z",
"Path": "docker-entrypoint.sh",
"Args": [
"apache2-foreground"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Pid": 894,
"ExitCode": 0,
"Error": "",
"StartedAt": "2022-12-26T20:26:21.348936894Z",
"FinishedAt": "2022-08-31T15:40:20.366876609Z"
},
"Image": "sha256:0a3bdd32ae210e34ed07cf49669dccbdb9deac143ebf27d5cc83136a0e3d9063",
"Name": "/wordpress",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"html:/var/www/html"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {
"80/tcp": [
{
"HostIp": "",
"HostPort": "80"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"IpcMode": "private",
"Cgroup": "",
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceWriteBps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "html",
"Source": "/var/lib/docker/volumes/html/_data",
"Destination": "/var/www/html",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "6ffb084b515c",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"80/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"WORDPRESS_DB_USER=root",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"PHPIZE_DEPS=autoconf \t\tdpkg-dev \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c",
"PHP_INI_DIR=/usr/local/etc/php",
"APACHE_CONFDIR=/etc/apache2",
"APACHE_ENVVARS=/etc/apache2/envvars",
"PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
"PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
"PHP_LDFLAGS=-Wl,-O1 -pie",
"GPG_KEYS=42670A7FE4D0441C8E4632349E4FDC074A4EF02D 5A52880781F755608BF815FC910DEB46F53EA312",
"PHP_VERSION=7.4.30",
"PHP_URL=https://www.php.net/distributions/php-7.4.30.tar.xz",
"PHP_ASC_URL=https://www.php.net/distributions/php-7.4.30.tar.xz.asc",
"PHP_SHA256=ea72a34f32c67e79ac2da7dfe96177f3c451c3eefae5810ba13312ed398ba70d"
],
"Cmd": [
"apache2-foreground"
],
"Volumes": {
"/var/www/html": {}
},
"WorkingDir": "/var/www/html",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "fa7c866c125b6bdbe158555fca49a8d871d32282817db2f396e0ff43d7a93eb4",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project.config_files": "docker-compose.yaml",
"com.docker.compose.service": "wordpress",
"com.docker.compose.version": "1.25.0"
},
"StopSignal": "SIGWINCH"
},
"NetworkSettings": {
"Bridge": "",
"HairpinMode": false,
"Ports": {
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "80"
}
]
},
"SandboxKey": "/var/run/docker/netns/9f28df1912da",
"EndpointID": "3928232aebd334a5aca3706dd3739f070e99bae3723e515a7792874a5f434bc0",
"Gateway": "172.17.0.1",
"GlobalIPv6PrefixLen": 0,
"IPPrefixLen": 16,
"IPv6Gateway": "",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Aliases": null,
"EndpointID": "3928232aebd334a5aca3706dd3739f070e99bae3723e515a7792874a5f434bc0",
"Gateway": "172.17.0.1",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null
}
}
}
}
]
``````

This gives us the necessary parameters to recreate the container, and the passwords to access the database. `docker inspect mariadb | grep IPAddress` gives us the IP Address to access MariaDB.

``````root@ip-172-31-19-232:~# docker exec -it wordpress mysql -u root -ppassword -h 172.17.0.2
Welcome to the MariaDB monitor.  Commands end with ; or \g.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

Database changed
Empty set (0.000 sec)

``````

So MariaDB is accessible fine from the wordpress container, just the hostname isn’t set up. There’s no `--link` parameter isn’t given. So let’s recreate the container:

``````# docker stop wordpress
# docker rm wordpress
# docker create \
--name wordpress \
-v html:/var/www/html \
-p 80:80 \
-e WORDPRESS_DB_HOST=mysql \
-e WORDPRESS_DB_NAME=wordpress \
-e WORDPRESS_DB_USER=root \
# docker start wordpress
``````

As a oneliner:

``````docker stop wordpress ; \
docker rm wordpress ; \
docker run -d \
--name wordpress \
-v html:/var/www/html \
-p 80:80 \
-e WORDPRESS_DB_HOST=mysql \
-e WORDPRESS_DB_NAME=wordpress \
-e WORDPRESS_DB_USER=root \
``````

## Singara

There’s a k3s Kubernetes install you can access with kubectl. The Kubernetes YAML manifests under /home/admin have been applied. The objective is to access from the host the “webapp” web server deployed and find what message it serves (it’s a name of a town or city btw). In order to pass the check, the webapp Docker container should not be run separately outside Kubernetes as a shortcut.

Let’s have a look at the cluster’s state:

``````root@ip-10-0-0-64:~# kubectl get -A pod
NAMESPACE     NAME                                      READY   STATUS             RESTARTS        AGE
kube-system   helm-install-traefik-crd-nml28            0/1     Completed          0               101d
kube-system   helm-install-traefik-z54r4                0/1     Completed          2               101d
web           webapp-deployment-666b67994b-5sffz        0/1     ImagePullBackOff   0               101d
kube-system   coredns-b96499967-scfhc                   1/1     Running            7 (101d ago)    101d
kube-system   local-path-provisioner-7b7dc8d6f5-r8777   1/1     Running            7 (101d ago)    101d
kube-system   metrics-server-668d979685-gkzrn           1/1     Running            11 (101d ago)   101d
kube-system   svclb-traefik-7cfc151c-hqm5p              2/2     Running            8 (101d ago)    101d
kube-system   traefik-7cd4fcff68-g8t6k                  1/1     Running            7 (101d ago)    101d
kube-system   svclb-traefik-7cfc151c-m6842              2/2     Running            2 (115s ago)    100d
``````

Apparantly, the desired container image is missing from the container runtime’s image store. The intro text hints at it being present in the Docker Inc. runtime that’s installed on the system though:

``````root@ip-10-0-0-64:/home/admin# docker images
REPOSITORY   TAG        IMAGE ID       CREATED        SIZE
webapp       latest     9c082e2983bc   3 months ago   135MB
python       3.7-slim   c1d0bab51bbf   3 months ago   123MB
registry     2          3a0f7b0a13ef   4 months ago   24.1MB
``````

There it is. But it’s not in k3s’ image store:

``````root@ip-10-0-0-64:~# k3s ctr i ls | grep webapp
root@ip-10-0-0-64:~#
``````

Let’s add it to k3s’ containerd:

``````root@ip-10-0-0-64:~# sudo docker save webapp:latest | sudo k3s ctr images import -
unpacking docker.io/library/webapp:latest (sha256:25b7b5c6f8ff5fc3b3b89bb5a5c1b96c619e4152275638aeaa82841b4c14ebf8)...done
root@ip-10-0-0-64:~# k3s ctr i ls | grep webapp
docker.io/library/webapp:latest                                                                                    application/vnd.docker.distribution.manifest.v2+json      sha256:25b7b5c6f8ff5fc3b3b89bb5a5c1b96c619e4152275638aeaa82841b4c14ebf8 84.1 MiB  linux/amd64                                                                                             io.cri-containerd.image=managed
``````

Excellent. But the pod is still in `ImagePullBackOff`:

``````root@ip-10-0-0-64:~# kubectl get -n web pod
webapp-deployment-666b67994b-5sffz   0/1     ImagePullBackOff   0          101d
``````

Let’s have a look at the manifests:

``````root@ip-10-0-0-64:~# cd ~admin
apiVersion: apps/v1
kind: Deployment
name: webapp-deployment
namespace: web
spec:
selector:
matchLabels:
app: webapp
replicas: 1
template:
labels:
app: webapp
spec:
containers:
- name: webapp
image: webapp
imagePullPolicy: Always
ports:
- containerPort: 8880
apiVersion: v1
kind: Service
name: webapp-service
namespace: web
spec:
type: NodePort
selector:
app.kubernetes.io/name: webapp
ports:
- port: 80
targetPort: 8888
nodePort: 30007
``````

The service doesn’t match the deployment, the labels don’t target the deployment, the port is 80 instead of 8888 and the service type doesn’t allocate the desired port on the host.

The deployment should not pull the image everytime and uses the wrong port, it should be 8888.

The manifests should look like this:

``````root@ip-10-0-0-64:/home/admin# cat deployment.yml
apiVersion: apps/v1
kind: Deployment
name: webapp-deployment
namespace: web
spec:
selector:
matchLabels:
app: webapp
replicas: 1
template:
labels:
app: webapp
spec:
containers:
- name: webapp
image: webapp
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8888
apiVersion: v1
kind: Service
name: webapp-service
namespace: web
spec:
selector:
app: webapp
ports:
- port: 8888
targetPort: 8888
nodePort: 30007
``````

Apply them with `kubectl apply -f deployment.yaml -f nodeport.yaml` and verify the results:

``````root@ip-10-0-0-64:/home/admin# kubectl -n web get svc -o wide
NAME             TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)          AGE    SELECTOR
webapp-service   LoadBalancer   10.43.35.97   172.31.33.52   8888:30007/TCP   101d   app=webapp
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them
-A CNI-DN-7a71a21266205d85a33bf -s 10.42.1.0/24 -p tcp -m tcp --dport 8888 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-7a71a21266205d85a33bf -s 127.0.0.1/32 -p tcp -m tcp --dport 8888 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-7a71a21266205d85a33bf -p tcp -m tcp --dport 8888 -j DNAT --to-destination 10.42.1.10:8888
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cbr0\" id: \"1c91e84359f8f6ac9bdcde61f61f3fedf231aedbc8ca174fa473231ec41298e0\"" -m multiport --dports 8888 -j CNI-DN-7a71a21266205d85a33bf
-A KUBE-SEP-72FXFZAWXXLVOFL6 -p tcp -m comment --comment "web/webapp-service" -m tcp -j DNAT --to-destination 10.42.1.9:8888
-A KUBE-SERVICES -d 10.43.35.97/32 -p tcp -m comment --comment "web/webapp-service cluster IP" -m tcp --dport 8888 -j KUBE-SVC-6PCM5WOO3Q3DGU2X
-A KUBE-SERVICES -d 172.31.33.52/32 -p tcp -m comment --comment "web/webapp-service loadbalancer IP" -m tcp --dport 8888 -j KUBE-EXT-6PCM5WOO3Q3DGU2X
-A KUBE-SVC-6PCM5WOO3Q3DGU2X ! -s 10.42.0.0/16 -d 10.43.35.97/32 -p tcp -m comment --comment "web/webapp-service cluster IP" -m tcp --dport 8888 -j KUBE-MARK-MASQ
-A KUBE-SVC-6PCM5WOO3Q3DGU2X -m comment --comment "web/webapp-service -> 10.42.1.9:8888" -j KUBE-SEP-72FXFZAWXXLVOFL6
``````

We’re done:

``````root@ip-10-0-0-64:/home/admin# curl localhost:8888
Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch
``````

## Karakorum

There’s a binary at `/home/admin/wtfit` that nobody knows how it works or what it does (“what the fun is this”). Someone remembers something about wtfit needing to communicate to a service in order to start. Run this wtfit program so it doesn’t exit with an error, fixing or working around things that you need but are broken in this server. (Note that you can open more than one web “terminal”).

``````admin@ip-172-31-42-20:~\$ file wtfit
wtfit: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=_iHDgQr9_DkcDPzFkdEV/KeopbEbY41Cs4Jx-plaD/bLd_UQ3kiokG80g9hhbY/qtKkikMAZmQbLl9qoMCF, not stripped
total 6.2M
bash: /usr/bin/chmod: Permission denied
``````

Wat? Chmod is broken? Great…

``````sudo /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/chmod +x /usr/bin/chmod
``````

STrace to the rescue…

``````admin@ip-172-31-42-20:~\$ strace -o xxx -f ./wtfit
ERROR: can't open config file
``````

Let’s look at `xxx`:

``````972   openat(AT_FDCWD, "/home/admin/wtfitconfig.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or d
irectory)
972   write(1, "ERROR: can't open config file\n", 30) = 30
``````

ok, again with the strace-and-looking-at-the-dump dance:

``````986   connect(3, {sa_family=AF_INET, sin_port=htons(7777), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...>
987   <... nanosleep resumed>NULL)      = 0
987   nanosleep({tv_sec=0, tv_nsec=320000},  <unfinished ...>
986   <... connect resumed>)            = -1 EINPROGRESS (Operation now in progress)
986   epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3781652200, u64=140337543081704}}) = 0
990   <... epoll_pwait resumed>[{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP|EPOLLRDHUP, {u32=3781652200, u64=140337543081704}}], 128, 29999, NULL, 580717888312) = 1
986   write(6, "\0", 1 <unfinished ...>
990   epoll_pwait(4,  <unfinished ...>
986   <... write resumed>)              = 1
990   <... epoll_pwait resumed>[{EPOLLIN, {u32=8851208, u64=8851208}}], 128, 29989, NULL, 580717888262) = 1
986   futex(0xc000034948, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
986   <... futex resumed>)              = 1
990   <... read resumed>"\0", 16)       = 1
989   <... futex resumed>)              = 0
987   <... nanosleep resumed>NULL)      = 0
986   getsockopt(3, SOL_SOCKET, SO_ERROR,  <unfinished ...>
990   futex(0xc000080148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
989   epoll_pwait(4,  <unfinished ...>
986   <... getsockopt resumed>[ECONNREFUSED], [4]) = 0
986   futex(0xc000080148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
987   nanosleep({tv_sec=0, tv_nsec=640000},  <unfinished ...>
986   <... futex resumed>)              = 1
990   <... futex resumed>)              = 0
986   epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc000093324 <unfinished ...>
990   futex(0xc000080148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
986   <... epoll_ctl resumed>)          = 0
986   close(3)                          = 0
986   futex(0xc000080148, FUTEX_WAKE_PRIVATE, 1) = 1
990   <... futex resumed>)              = 0
990   nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
986   write(1, "ERROR: can't connect to server\n", 31 <unfinished ...>
987   <... nanosleep resumed>NULL)      = 0
987   nanosleep({tv_sec=0, tv_nsec=1280000},  <unfinished ...>
986   <... write resumed>)              = 31
986   exit_group(1 <unfinished ...>
990   <... nanosleep resumed>NULL)      = 0
986   <... exit_group resumed>)         = ?
989   <... epoll_pwait resumed> <unfinished ...>) = ?
988   <... futex resumed>)              = ?
987   <... nanosleep resumed> <unfinished ...>) = ?
990   +++ exited with 1 +++
987   +++ exited with 1 +++
988   +++ exited with 1 +++
989   +++ exited with 1 +++
986   +++ exited with 1 +++
``````

So it wants to connect to `localhost:7777` but can’t, as there’s no service listening, as evidenced by `ss`. No mentions of the port in `/etc`.

Sigh so I guess we have to improvise a server. Let’s try with `Netcat`:

``````admin@ip-172-31-42-20:~\$ nc -p 7777 -l &
[1] 1042
admin@ip-172-31-42-20:~\$ strace -f -o xxx ./wtfit
GET / HTTP/1.1
Host: localhost:7777
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

^Z
[2]+  Stopped                 strace -f -o xxx ./wtfit
[2]+ strace -f -o xxx ./wtfit &
nc -p 7777 -l
^C
admin@ip-172-31-42-20:~\$ ERROR: can't connect to server

[2]+  Exit 1                  strace -f -o xxx ./wtfit
``````

So the binary sends an HTTP request to the endpoint in question and waits for an answer. So we need to point it to a web server, but we can’t install stuff. Python’s embedded webserver to the rescue:

``````admin@ip-172-31-42-20:~\$ python3 -m http.server --bind 127.0.0.1 7777 &
[1] 1056
admin@ip-172-31-42-20:~\$ Serving HTTP on 127.0.0.1 port 7777 (http://127.0.0.1:7777/) ...

admin@ip-172-31-42-20:~\$ strace -f -o xxx ./wtfit
127.0.0.1 - - [26/Dec/2022 21:35:23] "GET / HTTP/1.1" 200 -
OK.
``````

As a oneliner:

``````sudo /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/chmod +x /usr/bin/chmod ; \
cd ; \
chmod +x wtfit ; \
touch wtfitconfig.conf ; \
python3 -m http.server --bind localhost 7777 & ; \
./wtfit
``````

## Oaxaca

`lsof /home/admin/somefile` and `echo \$\$` reveals the bash we’re in has the file handle open. So we need to find out which one it is:

``````# Find the file descriptor
fd=find /proc/\$\$/fd -lname '/home/admin/somefile' | grep -oEe "[0-9]+\$"
# Close the file descriptor
exec \${fd}<&-
``````

As a oneliner:

``````exec \$(find /proc/\$\$/fd -lname '/home/admin/somefile' | grep -oEe "[0-9]+\$")<&-
``````

## Venice

Try and figure out if you are inside a container (like a Docker one for example) or inside a Virtual Machine (like in the other scenarios).

So we’re in a VM :-)

``````root@ip-172-31-38-151:/# dmesg | grep Hyper
[    0.000000] Hypervisor detected: KVM
``````
``````root@ip-172-31-38-151:/# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4UGO27476EXYY2UQSGBWL6EZC4:/var/lib/containers/storage/overlay/l/3JZQV4UY3FCO3W7SL3ERYENEZN:/var/lib/containers/storage/overlay/l/SCP4AZOKFN5HY4R5CQ5UVOYS7K:/var/lib/containers/storage/overlay/l/LPA46WOYFQ5ZHRJPEGNEEUCN36:/var/lib/containers/storage/overlay/l/WYTOBRCWJIZJALTLB3T5GBAQAB,upperdir=/var/lib/containers/storage/overlay/fb7c300d0cb0c61b15087365f7e0043090a7be0369605bf1d6efb574f78bd65a/diff,workdir=/var/lib/containers/storage/overlay/fb7c300d0cb0c61b15087365f7e0043090a7be0369605bf1d6efb574f78bd65a/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64000k)
tmpfs on /etc/hosts type tmpfs (rw,nosuid,nodev,noexec,relatime,size=46632k,mode=755)
tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,relatime,size=46632k,mode=755)
tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,noexec,relatime,size=46632k,mode=755)
tmpfs on /run/.containerenv type tmpfs (rw,nosuid,nodev,noexec,relatime,size=46632k,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /var/log/journal type tmpfs (rw,nosuid,nodev,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,relatime,nsdelegate,memory_recursiveprot)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13272)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
``````

The mountpoints `/` and `/run/.containerenv` heavily hint that this system is a container. As we’re able to call `poweroff` and it works, it’s likely that our container is privileged.

## Melbourne

There is a Python WSGI web application file at `/home/admin/wsgi.py`, the purpose of which is to serve the string “Hello, world!”. This file is served by a Gunicorn server which is fronted by an nginx server (both servers managed by systemd). So the flow of an HTTP request is: Web Client (curl) -> Nginx -> Gunicorn -> wsgi.py . The objective is to be able to curl the localhost (on default port :80) and get back “Hello, world!”, using the current setup.

Let’s see what we have there:

``````admin@ip-172-31-42-37:/\$ curl -s http://localhost
LISTEN       0             128                        0.0.0.0:22                      0.0.0.0:*           ino:10776 sk:1 cgroup:/system.slice/ssh.service <->
LISTEN       0             4096                             *:6767                          *:*           users:(("sadagent",pid=600,fd=7)) uid:1000 ino:10760 sk:2 cgroup:/system.slice/sadagent.service v6only:0 <->
LISTEN       0             4096                             *:8080                          *:*           users:(("gotty",pid=599,fd=6)) uid:1000 ino:11485 sk:3 cgroup:/system.slice/gotty.service v6only:0 <->
LISTEN       0             128                           [::]:22                         [::]:*           ino:10778 sk:4 cgroup:/system.slice/ssh.service v6only:1 <->
``````

Great. Nothing’s running. Let’s start Nginx then:

``````admin@ip-172-31-42-37:/\$ sudo systemctl enable --now nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx
● nginx.service - A high performance web server and a reverse proxy server
Active: active (running) since Tue 2022-12-27 19:06:49 UTC; 5s ago
Docs: man:nginx(8)
Process: 936 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, statu>
Process: 937 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCE>
Main PID: 938 (nginx)
Memory: 11.1M
CPU: 36ms
CGroup: /system.slice/nginx.service
├─938 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─939 nginx: worker process
└─940 nginx: worker process

Dec 27 19:06:49 ip-172-31-42-37 systemd[1]: Starting A high performance web server and a reverse proxy s>
Dec 27 19:06:49 ip-172-31-42-37 systemd[1]: Started A high performance web server and a reverse proxy se>
``````

Ok, nginx is up. So is gunicorn:

``````admin@ip-172-31-42-37:/\$ systemctl status gunicorn
● gunicorn.service - gunicorn daemon
Active: active (running) since Tue 2022-12-27 19:05:12 UTC; 6min ago
TriggeredBy: ● gunicorn.socket
Main PID: 615 (gunicorn)
Memory: 17.1M
CPU: 333ms
CGroup: /system.slice/gunicorn.service
├─615 /usr/bin/python3 /usr/local/bin/gunicorn --bind unix:/run/gunicorn.sock wsgi
└─670 /usr/bin/python3 /usr/local/bin/gunicorn --bind unix:/run/gunicorn.sock wsgi

Dec 27 19:05:12 ip-172-31-42-37 systemd[1]: Started gunicorn daemon.
Dec 27 19:05:13 ip-172-31-42-37 gunicorn[615]: [2022-12-27 19:05:13 +0000] [615] [INFO] Starting gunicorn 20.1.0
Dec 27 19:05:13 ip-172-31-42-37 gunicorn[615]: [2022-12-27 19:05:13 +0000] [615] [INFO] Listening at: unix:/run/gunicorn.sock (615)
Dec 27 19:05:13 ip-172-31-42-37 gunicorn[615]: [2022-12-27 19:05:13 +0000] [615] [INFO] Using worker: sync
Dec 27 19:05:13 ip-172-31-42-37 gunicorn[670]: [2022-12-27 19:05:13 +0000] [670] [INFO] Booting worker with pid: 670
``````

And it’s listening on a unix socket. So let’s see what it’s doing:

``````admin@ip-172-31-42-37:/\$ curl -s http://localhost
<html>
<body>
<hr><center>nginx/1.18.0</center>
</body>
</html>
``````

Nothing useful. Have a look at the config then:

``````admin@ip-172-31-42-37:/\$ cat /etc/nginx/sites-enabled/default
server {
listen 80;

location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.socket;
}
}
``````

That’s not what Nginx expects. Correct this and use the correct socket location given by `systemctl status gunicorn`’s output:

``````upstream gunicorn {
server unix:/run/gunicorn.sock;
}
server {
listen 80;

location / {
include proxy_params;
proxy_pass http://gunicorn;
}
}
``````

Now Nginx correctly sends Gunicorn’s output:

``````root@ip-172-31-42-1:~# curl -v http://localhost/
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: gunicorn
< Date: Tue, 27 Dec 2022 19:27:56 GMT
< Connection: close
< Content-Type: text/html
< Content-Length: 0
<
* Closing connection 0
``````

But it’s empty? Ok, so check its socket directly:

``````root@ip-172-31-42-1:~# curl -v --unix-socket /run/gunicorn.sock http://localhost/
*   Trying /run/gunicorn.sock:0...
* Connected to localhost (/run/gunicorn.sock) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: gunicorn
< Date: Tue, 27 Dec 2022 19:27:56 GMT
< Connection: close
< Content-Type: text/html
< Content-Length: 0
<
* Closing connection 0
``````

Hm. The response’s content length should read differently. What’s going on? We need to look at the source code:

``````def application(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/html'), ('Content-Length', 0)])
return [b'Hello, world!']
``````

The application should correctly set the payload’s content length – or leave it out to use chunked transfer encoding. Change the source, Luke:

``````def application(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/html')])
return [b'Hello, world!']
``````

Restart gunicorn and test:

``````root@ip-172-31-42-1:~# systemctl restart gunicorn
root@ip-172-31-42-1:~# curl -v --unix-socket /run/gunicorn.sock http://localhost/
*   Trying /run/gunicorn.sock:0...
* Connected to localhost (/run/gunicorn.sock) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: gunicorn
< Date: Tue, 27 Dec 2022 19:32:32 GMT
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html
<
* Closing connection 0
Hello, world!
``````

Excellent, we’re done.

As a short script:

``````cat > /etc/nginx/sites-enabled/default << EOF
upstream gunicorn {
server unix:/run/gunicorn.sock;
}
server {
listen 80;

location / {
include proxy_params;
proxy_pass http://gunicorn;
}
}
EOF
def application(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/html')])
return [b'Hello, world!']
EOF
systemctl restart gunicorn
systemctl enable --now nginx
``````

## Hong-Kong

(Similar to “Manhattan” scenario but harder). Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don’t need to know details about it (although it may help).

Postgres information: it’s a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Let’s have a looksie:

``````root@ip-172-31-25-11:/# systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
Active: active (exited) since Tue 2022-12-27 19:50:14 UTC; 2min 13s ago
Process: 595 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 595 (code=exited, status=0/SUCCESS)

Dec 27 19:50:14 ip-172-31-25-11 systemd[1]: Starting PostgreSQL RDBMS...
Dec 27 19:50:14 ip-172-31-25-11 systemd[1]: Started PostgreSQL RDBMS.
``````

Great. According to Debian’s docs, we’re supposed to use the `postgresql@.service` unit, matching the `/etc/postgresql/14/main` tree as mentioned:

``````root@ip-172-31-25-11:/# systemctl status postgresql@14-main
● postgresql@14-main.service - PostgreSQL Cluster 14-main
Active: failed (Result: protocol) since Tue 2022-12-27 20:29:00 UTC; 47s ago
Process: 581 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main start (code=exited, sta

Dec 27 20:29:00 ip-172-31-25-11 systemd[1]: Starting PostgreSQL Cluster 14-main...
Dec 27 20:29:00 ip-172-31-25-11 postgresql@14-main[581]: Error: /opt/pgdata/main is not accessible or doe
Dec 27 20:29:00 ip-172-31-25-11 systemd[1]: postgresql@14-main.service: Can't open PID file /run/postgres
Dec 27 20:29:00 ip-172-31-25-11 systemd[1]: postgresql@14-main.service: Failed with result 'protocol'.
Dec 27 20:29:00 ip-172-31-25-11 systemd[1]: Failed to start PostgreSQL Cluster 14-main.
``````

Ok, so the data is missing, `/opt/pgdata` is empty. Let’s look at `/etc/fstab`:

``````root@ip-172-31-25-11:/# cat /etc/fstab
# /etc/fstab: static file system information
UUID=5db68868-2d70-449f-8b1d-f3c769ec01c7 / ext4 rw,discard,errors=remount-ro,x-systemd.growfs 0 1
UUID=72C9-F191 /boot/efi vfat defaults 0 0
/dev/xvdb /opt/pgdata xfs defaults,nofail 0 0
``````

Verify that the disk is present:

``````root@ip-172-31-25-11:/# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1      259:0    0    8G  0 disk
nvme1n1      259:1    0    8G  0 disk
├─nvme1n1p1  259:2    0  7.9G  0 part /
├─nvme1n1p14 259:3    0    3M  0 part
└─nvme1n1p15 259:4    0  124M  0 part /boot/efi
root@ip-172-31-25-11:/# blkid /dev/nvme0n1
/dev/nvme0n1: UUID="9a2e1bf9-50e8-41a9-9f8c-5dd837869c50" TYPE="xfs"
``````

No it’s not. But `/dev/nvme0n1` is a likely candidate. Beware that `/etc/fstab` contains the `nofail` flag. You need to restart SystemD after changing `/etc/fstab`:

``````root@ip-172-31-25-11:/# cat /etc/fstab
# /etc/fstab: static file system information
UUID=5db68868-2d70-449f-8b1d-f3c769ec01c7 / ext4 rw,discard,errors=remount-ro,x-systemd.growfs 0 1
UUID=72C9-F191 /boot/efi vfat defaults 0 0
/dev/nvme0n1 /opt/pgdata xfs defaults,nofail 0 0
root@ip-172-31-25-11:/# mount /opt/pgdata
root@ip-172-31-25-11:/# ls /opt/pgdata
deleteme  file1.bk  file2.bk  file3.bk  main
root@ip-172-31-25-11:/# df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             224M     0  224M   0% /dev
tmpfs             47M  1.5M   45M   4% /run
/dev/nvme1n1p1   7.7G  1.2G  6.1G  17% /
tmpfs            233M     0  233M   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs            233M     0  233M   0% /sys/fs/cgroup
/dev/nvme1n1p15  124M  278K  124M   1% /boot/efi
/dev/nvme0n1     8.0G  8.0G   28K 100% /opt/pgdata
``````

Mounting works, but the disk is full as it was in Manhattan.

Purge the `*.bk` files from disk, restart the service and we’re done:

``````root@ip-172-31-25-11:/# rm -f /opt/pgdata/*.bk
root@ip-172-31-25-11:/# systemctl start postgresql@14-main
``````

## Lisbon

There’s an etcd server running on `https://localhost:2379`, get the value for the key “foo”, ie `etcdctl get foo` or `curl https://localhost:2379/v2/keys/foo`

``````Tue Jan  2 07:41:39 UTC 2024
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
how to fix it, please visit the web page mentioned above.
Error:  client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate has expired or is not yet valid: current time 2024-01-02T07:42:07Z is after 2023-01-30T00:02:48Z

error #0: x509: certificate has expired or is not yet valid: current time 2024-01-02T07:42:07Z is after 2023-01-30T00:02:48Z
``````

This server’s clock is off by a year - reset it:

``````admin@ip-10-0-0-196:/\$ sudo date -s "2023-01-02 08:30:00+0100"
<html>
<body>
<hr><center>nginx/1.18.0</center>
</body>
</html>
``````

Good, `curl` can access the host now. But that’s `nginx` that’s replying, not `etcd`?

``````
● nginx.service - A high performance web server and a reverse proxy server
Active: active (running) since Mon 2023-01-02 07:41:28 UTC; 6min ago
Docs: man:nginx(8)
Process: 581 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 601 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 610 (nginx)
Memory: 12.9M
CPU: 203ms
CGroup: /system.slice/nginx.service
├─610 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─611 nginx: worker process
└─612 nginx: worker process

Jan 02 07:41:27 ip-10-0-0-196 systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 02 07:41:28 ip-10-0-0-196 systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
Jan 02 07:41:28 ip-10-0-0-196 systemd[1]: Started A high performance web server and a reverse proxy server.
LISTEN         0              4096                       127.0.0.1:2379                      0.0.0.0:*            uid:108 ino:12125 sk:1 cgroup:/system.slice/etcd.service <->
LISTEN         0              4096                       127.0.0.1:2380                      0.0.0.0:*            uid:108 ino:12121 sk:2 cgroup:/system.slice/etcd.service <->
LISTEN         0              128                          0.0.0.0:22                        0.0.0.0:*            ino:11458 sk:3 cgroup:/system.slice/ssh.service <->
LISTEN         0              511                          0.0.0.0:443                       0.0.0.0:*            ino:11464 sk:4 cgroup:/system.slice/nginx.service <->
LISTEN         0              4096                               *:6767                            *:*            users:(("sadagent",pid=563,fd=7)) uid:1000 ino:10633 sk:5 cgroup:/system.slice/sadagent.service v6only:0 <->
LISTEN         0              4096                               *:8080                            *:*            users:(("gotty",pid=562,fd=6)) uid:1000 ino:10642 sk:6 cgroup:/system.slice/gotty.service v6only:0 <->
LISTEN         0              128                             [::]:22                           [::]:*            ino:11460 sk:7 cgroup:/system.slice/ssh.service v6only:1 <->
``````

Everything as it should be. What does `etcd` have to say?

``````admin@ip-10-0-0-196:/\$ systemctl status etcd
● etcd.service - etcd - highly-available key value store
Active: active (running) since Mon 2023-01-02 07:46:08 UTC; 4min 1s ago
Docs: https://etcd.io/docs
man:etcd
Main PID: 873 (etcd)
Memory: 7.8M
CPU: 2.050s
CGroup: /system.slice/etcd.service
└─873 /usr/bin/etcd --cert-file /etc/ssl/certs/localhost.crt --key-file /etc/ssl/certs/localhost.key --advertise-client-urls=https://localhost:2379 --listen-client-urls=https://localhost:2379

Jan 02 07:46:08 ip-10-0-0-196 etcd[873]: 8e9e05c52164694d became candidate at term 19
Jan 02 07:46:08 ip-10-0-0-196 etcd[873]: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 19
Jan 02 07:46:08 ip-10-0-0-196 etcd[873]: 8e9e05c52164694d became leader at term 19
Jan 02 07:46:08 ip-10-0-0-196 etcd[873]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 19
Jan 02 07:46:08 ip-10-0-0-196 etcd[873]: published {Name:ip-10-0-0-196 ClientURLs:[https://localhost:2379]} to cluster cdf818194e3a8c32
Jan 02 07:46:08 ip-10-0-0-196 etcd[873]: ready to serve client requests
Jan 02 07:46:08 ip-10-0-0-196 systemd[1]: Started etcd - highly-available key value store.
Jan 02 07:46:09 ip-10-0-0-196 etcd[873]: serving client requests on 127.0.0.1:2379
Jan 02 07:46:11 ip-10-0-0-196 etcd[873]: WARNING: 2023/01/02 07:46:11 grpc: addrConn.createTransport failed to connect to {localhost:2379  <nil> 0 <nil>}. Err :tls: use of closed connection. Reconnecting...
Jan 02 07:46:23 ip-10-0-0-196 etcd[873]: WARNING: 2023/01/02 07:46:23 grpc: addrConn.createTransport failed to connect to {localhost:2379  <nil> 0 <nil>}. Err :tls: use of closed connection. Reconnecting...
``````

`etcd` does get connections, but something doesn’t work - So what’s interfering? IPTables?

``````admin@ip-10-0-0-196:/\$ sudo iptables-save
# Generated by iptables-save v1.8.7 on Mon Jan  2 07:52:37 2023
*nat
:PREROUTING ACCEPT [13:1812]
:INPUT ACCEPT [11:660]
:OUTPUT ACCEPT [13:988]
:POSTROUTING ACCEPT [50:3208]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -o lo -p tcp -m tcp --dport 2379 -j REDIRECT --to-ports 443
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Jan  2 07:52:37 2023
# Generated by iptables-save v1.8.7 on Mon Jan  2 07:52:37 2023
*filter
:INPUT ACCEPT [2199:212561]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [1644:453532]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan  2 07:52:37 2023
``````

The `-A OUTPUT -o lo -p tcp -m tcp --dport 2379 -j REDIRECT --to-ports 443` line is the smoking gun right there. `sudo docker ps -a` verifies that Docker isn’t relevant to this exercise, so purge the IPTables NAT table:

``````admin@ip-10-0-0-196:/\$ sudo iptables -t nat -F
And presto, all is well with `etcd`!