Ein Befehl, ein simulierter Vorfall: von der ersten Prüfung bis zum Bericht — in 50 Sekunden.
Text-Version der Aufzeichnung
___ ____ ____ ____ ___ ____ _____
/ _ \| _ \/ ___| / ___/ _ \| _ \| ____| Linux server diagnostics
| | | | |_) \___ \| | | | | | |_) | _| services . network . storage . web
| |_| | __/ ___) | |__| |_| | __/| |___ database . mail . security . containers
\___/|_| |____/ \____\___/|_| |_____| read-only . evidence . report
v2.0.0 | host: demo-web-01 (SIMULATED) | mode: demo (read-only) | logs since: 24 hours ago
DEMO MODE - all results are SIMULATED
[ OK ] Read-only safety mode no changes to services, files or config
[ OK ] Privileges user
[ OK ] Loading modules system network storage performance logs web database mail security cloud
[ OK ] Knowledge base 36 log patterns
[ OK ] Correlation engine 13 rules armed
[ OK ] Secret redaction passwords, tokens, keys
[ OK ] Run directory reports/20261020-215822_DEMO-incident
Main menu
1) Full System Audit
2) Service Diagnostics (systemd, failed units, kernel, NTP)
3) Network & DNS (routes, DNS, TCP, HTTP, ports)
4) Disk & Memory (filesystems, inodes, RAM, swap, OOM)
5) Log Analysis (grouped errors, timeline, explanations)
6) Mail Server (queue, ports, MX/SPF/DMARC, port 25)
7) Database (PostgreSQL, MySQL/MariaDB)
8) Web Server (nginx/Apache, HTTP, TLS, 5xx)
9) Security & Access (SSH, logins, firewall, updates)
10) Performance (load, CPU, I/O wait, steal, PSI)
11) Cloud & Containers (Docker, Kubernetes, platform)
12) Generate Report
q) Quit
Choice: 1
────────────────────────────────────────────────────────────────────────
System & Service Diagnostics
────────────────────────────────────────────────────────────────────────
[INFO] Distribution: Ubuntu 22.04.4 LTS [SIMULATED]
[INFO] Uptime: 41d 3h [SIMULATED]
[WARNING] System state: degraded [SIMULATED]
↳ Why: systemd is up but at least one unit has failed
↳ Next: systemctl --failed
[CRITICAL] nginx.service: failed [SIMULATED]
↳ Why: Unit failed: crash, bad configuration, port conflict, missing dependency, full disk or OOM
↳ Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
↳ Investigating nginx: collecting status and recent journal entries...
↳ Evidence saved: evidence/service_nginx.txt (systemctl status + journalctl -u nginx (SIMULATED))
│ 2026-10-19T09:58:12+0200 demo-web-01 nginx[2211]: 2026/10/19 09:58:12 [crit] 2211#0: pwrite() "/var/lib/nginx/body/0000000012" failed (28: No space le
│ 2026-10-19T09:58:40+0200 demo-web-01 nginx[2211]: open() "/var/log/nginx/access.log" failed (28: No space left on device)
│ 2026-10-19T10:02:03+0200 demo-web-01 systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE
│ 2026-10-19T10:02:03+0200 demo-web-01 systemd[1]: nginx.service: Failed with result exit-code.
[OK] ssh service: active (running, enabled) [SIMULATED]
[OK] cron service: active (running, enabled) [SIMULATED]
[OK] Clock synchronised (NTP): yes (tz Europe/Rome) [SIMULATED]
💡 Did you know? Sysadmin haiku: It's not DNS. There's no way it's DNS. It was DNS.
────────────────────────────────────────────────────────────────────────
Network Diagnostics
────────────────────────────────────────────────────────────────────────
[OK] Global IP addresses: 1 (eth0 10.0.1.25/24) [SIMULATED]
[OK] Default route: via 10.0.1.1 dev eth0 [SIMULATED]
[OK] Configured nameservers: 10.0.0.2 [SIMULATED]
[CRITICAL] Resolve example.com: FAILED (timeout) [SIMULATED]
↳ Why: Name could not be resolved: resolver unreachable, misconfigured, or record missing
↳ Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
[CRITICAL] Nameserver 10.0.0.2 answers: no answer [SIMULATED]
↳ Why: This resolver does not answer: down or UDP/TCP 53 blocked
↳ Next: dig example.com @10.0.0.2 # check security group port 53
[CRITICAL] DNS resolution: 0/2 names resolved [SIMULATED]
↳ Why: No name resolves
↳ Next: resolvectl status
[OK] TCP 1.1.1.1:443: open (12ms) [SIMULATED]
[OK] Outbound IP connectivity: 1 IP endpoint(s) reachable [SIMULATED]
[INFO] Listening TCP ports: 22 3306 [SIMULATED]
[CRITICAL] HTTP port 80 listening: no [SIMULATED]
↳ Why: Nothing listens on port 80: the web server is not running
↳ Next: ss -tlnp | grep ':80'
💡 Did you know? Linux load average also counts tasks waiting on disk I/O: high load can mean slow storage, not busy CPUs.
────────────────────────────────────────────────────────────────────────
Disk & Memory Analysis
────────────────────────────────────────────────────────────────────────
[OK] Disk usage /: 54% (9.1 GiB free) [SIMULATED]
[CRITICAL] Disk usage /var: 96% (812 MiB free of 20.0 GiB) [SIMULATED] (threshold: warn 80% / crit 90%)
↳ Why: Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
↳ Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
↳ Largest directories in /var:
│ /var/log=14.2G /var/lib=3.9G /var/cache=0.8G
│ /var/log: nginx=11.8G journal=1.9G
[OK] Memory in use: 48% (2.0 GiB available of 3.8 GiB) [SIMULATED]
[OK] OOM killer events: 0 [SIMULATED]
💡 Did you know? HTTP 418 'I'm a teapot' comes from RFC 2324, an April Fools' joke from 1998.
────────────────────────────────────────────────────────────────────────
Performance Analysis
────────────────────────────────────────────────────────────────────────
[OK] Load per CPU core (5 min): 0.62 (load 2.71/2.48 on 4 cores) [SIMULATED]
[OK] CPU busy: 31% [SIMULATED]
[OK] CPU I/O wait: 2.1% [SIMULATED]
[OK] CPU steal (virtualisation): 0.4% [SIMULATED]
[OK] Pressure stall io (avg 60s): 1.8% of time [SIMULATED]
[OK] Zombie processes: 0 [SIMULATED]
💡 Did you know? Ping is not proof of life: many firewalls drop ICMP while TCP services work fine.
────────────────────────────────────────────────────────────────────────
Log Analysis
────────────────────────────────────────────────────────────────────────
[WARNING] Error lines in journal (since 24 hours ago): 10 [SIMULATED]
↳ Why: Repeated errors grouped below
↳ Next: journalctl -p err --since '24 hours ago'
↳ Most repeated messages (numbers normalised):
│ 3 app: ERROR could not resolve api.payments.example: Temporary failure in name resolution
│ 1 systemd: nginx.service: Main process exited, code=exited, status=<n>/FAILURE
│ 1 systemd: nginx.service: Failed with result exit-code.
│ 1 systemd: Failed to start A high performance web server.
Error timeline (per hour):
2026-10-19 09:00 ████████ 2
2026-10-19 10:00 ██████████████████████████████ 7
2026-10-19 11:00 ████ 1
↳ Peak: 2026-10-19T10:00 (7 events)
↳ Evidence saved: evidence/log_errors.txt (error lines (SIMULATED; note the redacted secrets))
[WARNING] Logs pattern 'unit_failed': 3 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
↳ Why: A systemd unit failed to start or exited with an error
↳ Next: systemctl --failed; journalctl -u <unit> -n 50
↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
│ 2026-10-19T10:02:03+0200 demo-web-01 systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE
[WARNING] Logs pattern 'dns_fail': 3 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
↳ Why: Name resolution failures: resolver unreachable/misconfigured or record missing
↳ Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
│ 2026-10-19T10:15:22+0200 demo-web-01 app[3120]: ERROR could not resolve api.payments.example: Temporary failure in name resolution
[CRITICAL] Logs pattern 'disk_full': 2 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
↳ Why: A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
↳ Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
│ 2026-10-19T09:58:12+0200 demo-web-01 nginx[2211]: 2026/10/19 09:58:12 [crit] 2211#0: pwrite() "/var/lib/nginx/body/0000000012" failed (28: No space le
[WARNING] Logs pattern 'timeout': 1 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
↳ Why: Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
↳ Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
│ 2026-10-19T10:16:52+0200 demo-web-01 app[3120]: ERROR request failed password=[REDACTED] token=[REDACTED] (connection timed out)
[WARNING] Logs pattern 'auth_fail': 1 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
↳ Why: Authentication failures: brute-force attempts or users with wrong credentials
↳ Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
│ 2026-10-19T11:01:09+0200 demo-web-01 sshd[4410]: Failed password for invalid user admin from 203.0.113.50 port 51122 ssh2
💡 Did you know? Sysadmin haiku: It's not DNS. There's no way it's DNS. It was DNS.
────────────────────────────────────────────────────────────────────────
Web Server Diagnostics
────────────────────────────────────────────────────────────────────────
[CRITICAL] nginx service: failed [SIMULATED]
↳ Why: The web server crashed or failed to start (config error, port conflict, full disk or OOM)
↳ Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
[OK] nginx -t (config test): syntax ok [SIMULATED]
↳ Config is valid: the start failure is not a syntax problem -> look at resources (disk)
[CRITICAL] Port 443 listening: no [SIMULATED]
↳ Why: A web server is installed but nothing listens on 443
↳ Next: ss -tlnp | grep ':443'
[CRITICAL] GET https://localhost/: connection failed (refused or unreachable) [SIMULATED]
↳ Why: No HTTP answer
↳ Next: curl -vk https://localhost/
[WARNING] TLS certificate shop.example.com:443: expires in 12 days (Oct 31 23:59:59 2026 GMT) [SIMULATED] (threshold: warn < 30d / crit < 7d)
↳ Why: Certificate close to expiry: renewal may be failing
↳ Next: certbot certificates; systemctl list-timers | grep -i certbot
💡 Did you know? Tux the penguin was drawn by Larry Ewing in 1996 using GIMP.
────────────────────────────────────────────────────────────────────────
Database Diagnostics
────────────────────────────────────────────────────────────────────────
[OK] postgresql@16-main service: active (running, enabled) [SIMULATED]
[OK] PostgreSQL accepting connections: yes [SIMULATED]
[WARNING] MySQL/MariaDB port 3306: listening on all interfaces (0.0.0.0) (suspected) [SIMULATED]
↳ Why: Database reachable from the network: verify firewall/security group allow only app servers
↳ Next: ufw status; security group rules # check bind-address
[OK] Data dir /var/lib/postgresql filesystem: 54% used [SIMULATED]
💡 Did you know? Linus Torvalds announced Linux on 25 Aug 1991 as 'just a hobby, won't be big and professional'.
────────────────────────────────────────────────────────────────────────
Mail Server Diagnostics
────────────────────────────────────────────────────────────────────────
[OK] postfix service: active (running, enabled) [SIMULATED]
[WARNING] Mail queue: 142 message(s) [SIMULATED] (threshold: warn 50 / crit 500)
↳ Why: Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
↳ Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
[WARNING] Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout (suspected) [SIMULATED]
↳ Why: Outbound port 25 blocked: common default at cloud providers
↳ Next: # check provider policy; use an authenticated relay on 587
↳ Limits: No email was sent during this test
[WARNING] Mail log pattern 'smtp_deferred': 1 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
↳ Why: Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
↳ Next: postqueue -p | tail # check outbound TCP/25 and the 'said:' reply in the log
↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
│ Oct 19 10:20:01 demo-web-01 postfix/smtp[5120]: 4F2A1C: to=<a@example.org>, relay=none, delay=3600, status=deferred (connect to mx.example.org[192.0.2
[OK] MX example-shop.com: mx1.example-shop.com [SIMULATED]
[WARNING] DMARC example-shop.com: missing (suspected) [SIMULATED]
↳ Why: No DMARC policy: large providers may reject
↳ Next: dig TXT _dmarc.example-shop.com
💡 Did you know? Linus Torvalds announced Linux on 25 Aug 1991 as 'just a hobby, won't be big and professional'.
────────────────────────────────────────────────────────────────────────
Security & Access Checks
────────────────────────────────────────────────────────────────────────
[OK] SSH PermitRootLogin: prohibit-password [SIMULATED]
[WARNING] SSH PasswordAuthentication: yes (suspected) [SIMULATED]
↳ Why: Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
↳ Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs
[CRITICAL] Failed SSH logins (since 24 hours ago): 1843 [SIMULATED] (threshold: warn 20 / crit 500)
↳ Why: Password-guessing against SSH
↳ Next: # consider fail2ban; restrict port 22
│ 912 203.0.113.50
│ 455 198.51.100.23
│ 201 192.0.2.77
[OK] Firewall (ufw): active, 4 allow rules [SIMULATED]
[WARNING] Pending updates (apt): 37 total, 23 security [SIMULATED]
↳ Why: Known vulnerabilities fixed upstream but not installed
↳ Next: apt list --upgradable | grep security
[OK] Accounts with UID 0 besides root: none [SIMULATED]
💡 Did you know? Tux the penguin was drawn by Larry Ewing in 1996 using GIMP.
────────────────────────────────────────────────────────────────────────
Cloud & Container Checks
────────────────────────────────────────────────────────────────────────
[INFO] Platform: OpenStack (Nova) [SIMULATED]
[OK] Docker daemon: running (v27.3.1) [SIMULATED]
[CRITICAL] Containers restarting: queue-worker [SIMULATED]
↳ Why: Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
↳ Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
↳ Evidence saved: evidence/container_queue-worker.txt (docker logs --tail 20 queue-worker (SIMULATED))
│ 2026-10-19 10:31:02 ERROR cannot write /var/lib/worker/tmp/job-8812: No space left on device
│ 2026-10-19 10:31:02 worker exited with code 1
[OK] Containers running: 11 [SIMULATED]
[INFO] Kubernetes: kubectl not configured on this host [SIMULATED]
────────────────────────────────────────────────────────────────────────
Anomaly analysis & intervention priorities
────────────────────────────────────────────────────────────────────────
Correlated findings (deterministic rules - correlation, not proof)
P1 Full filesystem is the likely trigger of the service failures
Evidence: Disk usage /var: 96% (812 MiB free of 20.0 GiB);nginx service: failed;log pattern 'No space left on device' present
Meaning: Logs contain 'No space left on device' and a filesystem is above the critical threshold while services are down. This is strong supporting evidence, but confirm in each unit's journal before acting.
Next: # Free space first (logs rotation, old releases, journalctl --vacuum-size after approval), then restart the affected services in a maintenance window.
P1 DNS-layer failure: IP connectivity works, name resolution does not
Evidence: Resolve example.com: FAILED (timeout);Nameserver 10.0.0.2 answers: no answer;DNS resolution: 0/2 names resolved;Outbound IP connectivity: 1 IP endpoint(s) reachable
Meaning: TCP connections to IP addresses succeed, so routing/firewall egress is fine; the fault is isolated to resolver configuration or the upstream DNS servers.
Next: cat /etc/resolv.conf; resolvectl status; dig example.com @<each-nameserver> # check that UDP+TCP/53 to the resolver is allowed (security group/NACL)
P1 nginx is down: closed HTTP ports and failing HTTP checks are consequences of the same fault
Evidence: nginx service: failed;HTTP port 80 listening: no;Port 443 listening: no;GET https://localhost/: connection failed (refused or unreachable)
Meaning: The listener is missing because the process is not running. Fix the service start-up error first; the HTTP symptoms should then clear.
Next: nginx -t; journalctl -u nginx -n 50; ss -tlnp | grep -E ':80|:443' # config test; port conflict?
P2 SSH exposed to password guessing
Evidence: Failed SSH logins (since 24 hours ago): 1843;SSH PasswordAuthentication: yes
Meaning: Many failed logins while password authentication is enabled increases the risk of credential compromise.
Next: # Restrict port 22 (security group/VPN), prefer key-only auth, consider fail2ban (changes require approval)
P2 Deferred mail is consistent with blocked outbound SMTP (port 25)
Evidence: Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout
Meaning: Many cloud providers block or throttle outbound TCP/25 by default; remote MX servers are then unreachable and mail stays deferred.
Next: # Verify provider policy, request unblock or use an authenticated relay on 587
Anomalies by priority (25)
P1 nginx service = failed [web]
Why: The web server crashed or failed to start (config error, port conflict, full disk or OOM)
Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
P1 Resolve example.com = FAILED (timeout) [network]
Why: Name could not be resolved: resolver unreachable, misconfigured, or record missing
Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
P1 Nameserver 10.0.0.2 answers = no answer [network]
Why: This resolver does not answer: down or UDP/TCP 53 blocked
Next: dig example.com @10.0.0.2 # check security group port 53
P1 DNS resolution = 0/2 names resolved [network]
Why: No name resolves
Next: resolvectl status
P1 HTTP port 80 listening = no [network]
Why: Nothing listens on port 80: the web server is not running
Next: ss -tlnp | grep ':80'
P1 Disk usage /var = 96% (812 MiB free of 20.0 GiB) [storage]
Why: Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
P1 Logs pattern 'disk_full' = 2 occurrence(s) [logs]
Why: A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
P1 Port 443 listening = no [web]
Why: A web server is installed but nothing listens on 443
Next: ss -tlnp | grep ':443'
P1 GET https://localhost/ = connection failed (refused or unreachable) [web]
Why: No HTTP answer
Next: curl -vk https://localhost/
P1 Failed SSH logins (since 24 hours ago) = 1843 [security]
Why: Password-guessing against SSH
Next: # consider fail2ban; restrict port 22
P1 Containers restarting = queue-worker [cloud]
Why: Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
P2 System state = degraded [system]
Why: systemd is up but at least one unit has failed
Next: systemctl --failed
P2 Error lines in journal (since 24 hours ago) = 10 [logs]
Why: Repeated errors grouped below
Next: journalctl -p err --since '24 hours ago'
P2 Logs pattern 'unit_failed' = 3 occurrence(s) [logs]
Why: A systemd unit failed to start or exited with an error
Next: systemctl --failed; journalctl -u <unit> -n 50
P2 Logs pattern 'dns_fail' = 3 occurrence(s) [logs]
Why: Name resolution failures: resolver unreachable/misconfigured or record missing
Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
P2 Logs pattern 'timeout' = 1 occurrence(s) [logs]
Why: Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
P2 Logs pattern 'auth_fail' = 1 occurrence(s) [logs]
Why: Authentication failures: brute-force attempts or users with wrong credentials
Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
P2 TLS certificate shop.example.com:443 = expires in 12 days (Oct 31 23:59:59 2026 GMT) [web]
Why: Certificate close to expiry: renewal may be failing
Next: certbot certificates; systemctl list-timers | grep -i certbot
P2 Mail queue = 142 message(s) [mail]
Why: Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
P2 Mail log pattern 'smtp_deferred' = 1 occurrence(s) [mail]
Why: Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
Next: postqueue -p | tail # check outbound TCP/25 and the 'said:' reply in the log
P2 Pending updates (apt) = 37 total, 23 security [security]
Why: Known vulnerabilities fixed upstream but not installed
Next: apt list --upgradable | grep security
P3 MySQL/MariaDB port 3306 = listening on all interfaces (0.0.0.0) (suspected) [database]
Why: Database reachable from the network: verify firewall/security group allow only app servers
Next: ufw status; security group rules # check bind-address
P3 Outbound SMTP to gmail-smtp-in.l.google.com:25 = timeout (suspected) [mail]
Why: Outbound port 25 blocked: common default at cloud providers
Next: # check provider policy; use an authenticated relay on 587
P3 DMARC example-shop.com = missing (suspected) [mail]
Why: No DMARC policy: large providers may reject
Next: dig TXT _dmarc.example-shop.com
P3 SSH PasswordAuthentication = yes (suspected) [security]
Why: Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs
────────────────────────────────────────────────────────────────────────
Run summary
────────────────────────────────────────────────────────────────────────
SIMULATED DATA - demo mode
Host: demo-web-01 (SIMULATED) Log window: since 24 hours ago
OK 28 WARNING 14 CRITICAL 12 UNKNOWN 0 INFO 5
Health score [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100
Verdict CRITICAL - immediate action required
Report: ~/opscope/reports/20261020-215822_DEMO-incident/report.md
Main menu
1) Full System Audit
2) Service Diagnostics (systemd, failed units, kernel, NTP)
3) Network & DNS (routes, DNS, TCP, HTTP, ports)
4) Disk & Memory (filesystems, inodes, RAM, swap, OOM)
5) Log Analysis (grouped errors, timeline, explanations)
6) Mail Server (queue, ports, MX/SPF/DMARC, port 25)
7) Database (PostgreSQL, MySQL/MariaDB)
8) Web Server (nginx/Apache, HTTP, TLS, 5xx)
9) Security & Access (SSH, logins, firewall, updates)
10) Performance (load, CPU, I/O wait, steal, PSI)
11) Cloud & Containers (Docker, Kubernetes, platform)
12) Generate Report
q) Quit
Choice: 12
────────────────────────────────────────────────────────────────────────
Anomaly analysis & intervention priorities
────────────────────────────────────────────────────────────────────────
Correlated findings (deterministic rules - correlation, not proof)
P1 Full filesystem is the likely trigger of the service failures
Evidence: Disk usage /var: 96% (812 MiB free of 20.0 GiB);nginx service: failed;log pattern 'No space left on device' present
Meaning: Logs contain 'No space left on device' and a filesystem is above the critical threshold while services are down. This is strong supporting evidence, but confirm in each unit's journal before acting.
Next: # Free space first (logs rotation, old releases, journalctl --vacuum-size after approval), then restart the affected services in a maintenance window.
P1 DNS-layer failure: IP connectivity works, name resolution does not
Evidence: Resolve example.com: FAILED (timeout);Nameserver 10.0.0.2 answers: no answer;DNS resolution: 0/2 names resolved;Outbound IP connectivity: 1 IP endpoint(s) reachable
Meaning: TCP connections to IP addresses succeed, so routing/firewall egress is fine; the fault is isolated to resolver configuration or the upstream DNS servers.
Next: cat /etc/resolv.conf; resolvectl status; dig example.com @<each-nameserver> # check that UDP+TCP/53 to the resolver is allowed (security group/NACL)
P1 nginx is down: closed HTTP ports and failing HTTP checks are consequences of the same fault
Evidence: nginx service: failed;HTTP port 80 listening: no;Port 443 listening: no;GET https://localhost/: connection failed (refused or unreachable)
Meaning: The listener is missing because the process is not running. Fix the service start-up error first; the HTTP symptoms should then clear.
Next: nginx -t; journalctl -u nginx -n 50; ss -tlnp | grep -E ':80|:443' # config test; port conflict?
P2 SSH exposed to password guessing
Evidence: Failed SSH logins (since 24 hours ago): 1843;SSH PasswordAuthentication: yes
Meaning: Many failed logins while password authentication is enabled increases the risk of credential compromise.
Next: # Restrict port 22 (security group/VPN), prefer key-only auth, consider fail2ban (changes require approval)
P2 Deferred mail is consistent with blocked outbound SMTP (port 25)
Evidence: Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout
Meaning: Many cloud providers block or throttle outbound TCP/25 by default; remote MX servers are then unreachable and mail stays deferred.
Next: # Verify provider policy, request unblock or use an authenticated relay on 587
Anomalies by priority (25)
P1 nginx service = failed [web]
Why: The web server crashed or failed to start (config error, port conflict, full disk or OOM)
Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
P1 Resolve example.com = FAILED (timeout) [network]
Why: Name could not be resolved: resolver unreachable, misconfigured, or record missing
Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
P1 Nameserver 10.0.0.2 answers = no answer [network]
Why: This resolver does not answer: down or UDP/TCP 53 blocked
Next: dig example.com @10.0.0.2 # check security group port 53
P1 DNS resolution = 0/2 names resolved [network]
Why: No name resolves
Next: resolvectl status
P1 HTTP port 80 listening = no [network]
Why: Nothing listens on port 80: the web server is not running
Next: ss -tlnp | grep ':80'
P1 Disk usage /var = 96% (812 MiB free of 20.0 GiB) [storage]
Why: Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
P1 Logs pattern 'disk_full' = 2 occurrence(s) [logs]
Why: A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
P1 Port 443 listening = no [web]
Why: A web server is installed but nothing listens on 443
Next: ss -tlnp | grep ':443'
P1 GET https://localhost/ = connection failed (refused or unreachable) [web]
Why: No HTTP answer
Next: curl -vk https://localhost/
P1 Failed SSH logins (since 24 hours ago) = 1843 [security]
Why: Password-guessing against SSH
Next: # consider fail2ban; restrict port 22
P1 Containers restarting = queue-worker [cloud]
Why: Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
P2 System state = degraded [system]
Why: systemd is up but at least one unit has failed
Next: systemctl --failed
P2 Error lines in journal (since 24 hours ago) = 10 [logs]
Why: Repeated errors grouped below
Next: journalctl -p err --since '24 hours ago'
P2 Logs pattern 'unit_failed' = 3 occurrence(s) [logs]
Why: A systemd unit failed to start or exited with an error
Next: systemctl --failed; journalctl -u <unit> -n 50
P2 Logs pattern 'dns_fail' = 3 occurrence(s) [logs]
Why: Name resolution failures: resolver unreachable/misconfigured or record missing
Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
P2 Logs pattern 'timeout' = 1 occurrence(s) [logs]
Why: Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
P2 Logs pattern 'auth_fail' = 1 occurrence(s) [logs]
Why: Authentication failures: brute-force attempts or users with wrong credentials
Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
P2 TLS certificate shop.example.com:443 = expires in 12 days (Oct 31 23:59:59 2026 GMT) [web]
Why: Certificate close to expiry: renewal may be failing
Next: certbot certificates; systemctl list-timers | grep -i certbot
P2 Mail queue = 142 message(s) [mail]
Why: Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
P2 Mail log pattern 'smtp_deferred' = 1 occurrence(s) [mail]
Why: Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
Next: postqueue -p | tail # check outbound TCP/25 and the 'said:' reply in the log
P2 Pending updates (apt) = 37 total, 23 security [security]
Why: Known vulnerabilities fixed upstream but not installed
Next: apt list --upgradable | grep security
P3 MySQL/MariaDB port 3306 = listening on all interfaces (0.0.0.0) (suspected) [database]
Why: Database reachable from the network: verify firewall/security group allow only app servers
Next: ufw status; security group rules # check bind-address
P3 Outbound SMTP to gmail-smtp-in.l.google.com:25 = timeout (suspected) [mail]
Why: Outbound port 25 blocked: common default at cloud providers
Next: # check provider policy; use an authenticated relay on 587
P3 DMARC example-shop.com = missing (suspected) [mail]
Why: No DMARC policy: large providers may reject
Next: dig TXT _dmarc.example-shop.com
P3 SSH PasswordAuthentication = yes (suspected) [security]
Why: Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs
────────────────────────────────────────────────────────────────────────
Run summary
────────────────────────────────────────────────────────────────────────
SIMULATED DATA - demo mode
Host: demo-web-01 (SIMULATED) Log window: since 24 hours ago
OK 28 WARNING 14 CRITICAL 12 UNKNOWN 0 INFO 5
Health score [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100
Verdict CRITICAL - immediate action required
Report: ~/opscope/reports/20261020-215822_DEMO-incident/report.md
Main menu
1) Full System Audit
2) Service Diagnostics (systemd, failed units, kernel, NTP)
3) Network & DNS (routes, DNS, TCP, HTTP, ports)
4) Disk & Memory (filesystems, inodes, RAM, swap, OOM)
5) Log Analysis (grouped errors, timeline, explanations)
6) Mail Server (queue, ports, MX/SPF/DMARC, port 25)
7) Database (PostgreSQL, MySQL/MariaDB)
8) Web Server (nginx/Apache, HTTP, TLS, 5xx)
9) Security & Access (SSH, logins, firewall, updates)
10) Performance (load, CPU, I/O wait, steal, PSI)
11) Cloud & Containers (Docker, Kubernetes, platform)
12) Generate Report
q) Quit
Choice: q
────────────────────────────────────────────────────────────────────────
Anomaly analysis & intervention priorities
────────────────────────────────────────────────────────────────────────
Correlated findings (deterministic rules - correlation, not proof)
P1 Full filesystem is the likely trigger of the service failures
Evidence: Disk usage /var: 96% (812 MiB free of 20.0 GiB);nginx service: failed;log pattern 'No space left on device' present
Meaning: Logs contain 'No space left on device' and a filesystem is above the critical threshold while services are down. This is strong supporting evidence, but confirm in each unit's journal before acting.
Next: # Free space first (logs rotation, old releases, journalctl --vacuum-size after approval), then restart the affected services in a maintenance window.
P1 DNS-layer failure: IP connectivity works, name resolution does not
Evidence: Resolve example.com: FAILED (timeout);Nameserver 10.0.0.2 answers: no answer;DNS resolution: 0/2 names resolved;Outbound IP connectivity: 1 IP endpoint(s) reachable
Meaning: TCP connections to IP addresses succeed, so routing/firewall egress is fine; the fault is isolated to resolver configuration or the upstream DNS servers.
Next: cat /etc/resolv.conf; resolvectl status; dig example.com @<each-nameserver> # check that UDP+TCP/53 to the resolver is allowed (security group/NACL)
P1 nginx is down: closed HTTP ports and failing HTTP checks are consequences of the same fault
Evidence: nginx service: failed;HTTP port 80 listening: no;Port 443 listening: no;GET https://localhost/: connection failed (refused or unreachable)
Meaning: The listener is missing because the process is not running. Fix the service start-up error first; the HTTP symptoms should then clear.
Next: nginx -t; journalctl -u nginx -n 50; ss -tlnp | grep -E ':80|:443' # config test; port conflict?
P2 SSH exposed to password guessing
Evidence: Failed SSH logins (since 24 hours ago): 1843;SSH PasswordAuthentication: yes
Meaning: Many failed logins while password authentication is enabled increases the risk of credential compromise.
Next: # Restrict port 22 (security group/VPN), prefer key-only auth, consider fail2ban (changes require approval)
P2 Deferred mail is consistent with blocked outbound SMTP (port 25)
Evidence: Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout
Meaning: Many cloud providers block or throttle outbound TCP/25 by default; remote MX servers are then unreachable and mail stays deferred.
Next: # Verify provider policy, request unblock or use an authenticated relay on 587
Anomalies by priority (25)
P1 nginx service = failed [web]
Why: The web server crashed or failed to start (config error, port conflict, full disk or OOM)
Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
P1 Resolve example.com = FAILED (timeout) [network]
Why: Name could not be resolved: resolver unreachable, misconfigured, or record missing
Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
P1 Nameserver 10.0.0.2 answers = no answer [network]
Why: This resolver does not answer: down or UDP/TCP 53 blocked
Next: dig example.com @10.0.0.2 # check security group port 53
P1 DNS resolution = 0/2 names resolved [network]
Why: No name resolves
Next: resolvectl status
P1 HTTP port 80 listening = no [network]
Why: Nothing listens on port 80: the web server is not running
Next: ss -tlnp | grep ':80'
P1 Disk usage /var = 96% (812 MiB free of 20.0 GiB) [storage]
Why: Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
P1 Logs pattern 'disk_full' = 2 occurrence(s) [logs]
Why: A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
P1 Port 443 listening = no [web]
Why: A web server is installed but nothing listens on 443
Next: ss -tlnp | grep ':443'
P1 GET https://localhost/ = connection failed (refused or unreachable) [web]
Why: No HTTP answer
Next: curl -vk https://localhost/
P1 Failed SSH logins (since 24 hours ago) = 1843 [security]
Why: Password-guessing against SSH
Next: # consider fail2ban; restrict port 22
P1 Containers restarting = queue-worker [cloud]
Why: Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
P2 System state = degraded [system]
Why: systemd is up but at least one unit has failed
Next: systemctl --failed
P2 Error lines in journal (since 24 hours ago) = 10 [logs]
Why: Repeated errors grouped below
Next: journalctl -p err --since '24 hours ago'
P2 Logs pattern 'unit_failed' = 3 occurrence(s) [logs]
Why: A systemd unit failed to start or exited with an error
Next: systemctl --failed; journalctl -u <unit> -n 50
P2 Logs pattern 'dns_fail' = 3 occurrence(s) [logs]
Why: Name resolution failures: resolver unreachable/misconfigured or record missing
Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
P2 Logs pattern 'timeout' = 1 occurrence(s) [logs]
Why: Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
P2 Logs pattern 'auth_fail' = 1 occurrence(s) [logs]
Why: Authentication failures: brute-force attempts or users with wrong credentials
Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
P2 TLS certificate shop.example.com:443 = expires in 12 days (Oct 31 23:59:59 2026 GMT) [web]
Why: Certificate close to expiry: renewal may be failing
Next: certbot certificates; systemctl list-timers | grep -i certbot
P2 Mail queue = 142 message(s) [mail]
Why: Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
P2 Mail log pattern 'smtp_deferred' = 1 occurrence(s) [mail]
Why: Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
Next: postqueue -p | tail # check outbound TCP/25 and the 'said:' reply in the log
P2 Pending updates (apt) = 37 total, 23 security [security]
Why: Known vulnerabilities fixed upstream but not installed
Next: apt list --upgradable | grep security
P3 MySQL/MariaDB port 3306 = listening on all interfaces (0.0.0.0) (suspected) [database]
Why: Database reachable from the network: verify firewall/security group allow only app servers
Next: ufw status; security group rules # check bind-address
P3 Outbound SMTP to gmail-smtp-in.l.google.com:25 = timeout (suspected) [mail]
Why: Outbound port 25 blocked: common default at cloud providers
Next: # check provider policy; use an authenticated relay on 587
P3 DMARC example-shop.com = missing (suspected) [mail]
Why: No DMARC policy: large providers may reject
Next: dig TXT _dmarc.example-shop.com
P3 SSH PasswordAuthentication = yes (suspected) [security]
Why: Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs
────────────────────────────────────────────────────────────────────────
Run summary
────────────────────────────────────────────────────────────────────────
SIMULATED DATA - demo mode
Host: demo-web-01 (SIMULATED) Log window: since 24 hours ago
OK 28 WARNING 14 CRITICAL 12 UNKNOWN 0 INFO 5
Health score [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100
Verdict CRITICAL - immediate action required
Report: ~/opscope/reports/20261020-215822_DEMO-incident/report.md