DE EN IT

~/lab / opscope

OPSCOPE

One command, one simulated incident: from the first check to the report — in 50 seconds.

Bash · read-only · 10 modules · simulated data · 2026-10-20

← Back to the labGet in touch
daniele@demo-web-01: ~/opscope — ./opscope.sh --demo
daniele@mikweb: ~/lab — less opscope/README.md

What OPSCOPE does

A read-only Bash toolkit for diagnosing Linux servers. One command runs the checks, interprets the results (OK / WARNING / CRITICAL / UNKNOWN), collects evidence, correlates symptoms and writes a report.

What happens in the recording

  • A simulated incident on a demo server (demo-web-01): the filesystem is full, nginx has failed, the DNS resolver does not answer.
  • For every finding: the cause (Why), the next command to copy (Next) and the limits of the statement (Limits) — read-only commands only.
  • At the end deterministic rules connect the symptoms, for example “Full filesystem is the likely trigger of the service failures” — explicitly as correlation, not proof.
  • Summary with health score and verdict; the report (Markdown) contains the collected evidence.

Principles

  • Read-only: nothing on the system is changed; it only writes to its own report directory.
  • No mandatory dependencies: Bash and standard tools; if a tool is missing the result is UNKNOWN with the reason and the analysis continues.
  • Timeouts on every command, no hangs on network or stale mounts.
  • Secrets (passwords, tokens, keys) are redacted in output, logs and evidence.
  • It never sends e-mail; the SMTP test only reads the server greeting.

Modules

Services (systemd, failed units, NTP) · Network & DNS · Disk & memory · Log analysis · Mail (Postfix/Exim/Dovecot, SPF/DMARC, queue) · Database · Web server (nginx/Apache, TLS) · Security (SSH, logins, firewall, updates) · Performance (load, I/O wait, PSI) · Cloud & containers (Docker, Kubernetes).

Status

Work in progress, the source code is private. All data in the recording is simulated (demo mode): no real servers, no customer data. Interested in talking about it? Write to me

Text version of the recording
   ___  ____  ____   ____ ___  ____  _____
  / _ \|  _ \/ ___| / ___/ _ \|  _ \| ____|   Linux server diagnostics
 | | | | |_) \___ \| |  | | | | |_) |  _|     services . network . storage . web
 | |_| |  __/ ___) | |__| |_| |  __/| |___    database . mail . security . containers
  \___/|_|   |____/ \____\___/|_|   |_____|   read-only . evidence . report
  v2.0.0  |  host: demo-web-01 (SIMULATED)  |  mode: demo (read-only)  |  logs since: 24 hours ago
   DEMO MODE - all results are SIMULATED

  [ OK ] Read-only safety mode              no changes to services, files or config
  [ OK ] Privileges                         user
  [ OK ] Loading modules                    system network storage performance logs web database mail security cloud
  [ OK ] Knowledge base                     36 log patterns
  [ OK ] Correlation engine                 13 rules armed
  [ OK ] Secret redaction                   passwords, tokens, keys
  [ OK ] Run directory                      reports/20261020-215822_DEMO-incident

  Main menu
    1) Full System Audit
    2) Service Diagnostics  (systemd, failed units, kernel, NTP)
    3) Network & DNS        (routes, DNS, TCP, HTTP, ports)
    4) Disk & Memory        (filesystems, inodes, RAM, swap, OOM)
    5) Log Analysis         (grouped errors, timeline, explanations)
    6) Mail Server          (queue, ports, MX/SPF/DMARC, port 25)
    7) Database             (PostgreSQL, MySQL/MariaDB)
    8) Web Server           (nginx/Apache, HTTP, TLS, 5xx)
    9) Security & Access    (SSH, logins, firewall, updates)
   10) Performance          (load, CPU, I/O wait, steal, PSI)
   11) Cloud & Containers   (Docker, Kubernetes, platform)
   12) Generate Report
    q) Quit
  Choice: 1

────────────────────────────────────────────────────────────────────────
  System & Service Diagnostics
────────────────────────────────────────────────────────────────────────
  [INFO]     Distribution: Ubuntu 22.04.4 LTS [SIMULATED]
  [INFO]     Uptime: 41d 3h [SIMULATED]
  [WARNING]  System state: degraded [SIMULATED]
             ↳ Why:    systemd is up but at least one unit has failed
             ↳ Next:   systemctl --failed
  [CRITICAL] nginx.service: failed [SIMULATED]
             ↳ Why:    Unit failed: crash, bad configuration, port conflict, missing dependency, full disk or OOM
             ↳ Next:   systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
  ↳ Investigating nginx: collecting status and recent journal entries...
  ↳ Evidence saved: evidence/service_nginx.txt  (systemctl status + journalctl -u nginx (SIMULATED))
             │ 2026-10-19T09:58:12+0200 demo-web-01 nginx[2211]: 2026/10/19 09:58:12 [crit] 2211#0: pwrite() "/var/lib/nginx/body/0000000012" failed (28: No space le
             │ 2026-10-19T09:58:40+0200 demo-web-01 nginx[2211]: open() "/var/log/nginx/access.log" failed (28: No space left on device)
             │ 2026-10-19T10:02:03+0200 demo-web-01 systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE
             │ 2026-10-19T10:02:03+0200 demo-web-01 systemd[1]: nginx.service: Failed with result exit-code.
  [OK]       ssh service: active (running, enabled) [SIMULATED]
  [OK]       cron service: active (running, enabled) [SIMULATED]
  [OK]       Clock synchronised (NTP): yes (tz Europe/Rome) [SIMULATED]

  💡 Did you know? Sysadmin haiku: It's not DNS. There's no way it's DNS. It was DNS.

────────────────────────────────────────────────────────────────────────
  Network Diagnostics
────────────────────────────────────────────────────────────────────────
  [OK]       Global IP addresses: 1 (eth0 10.0.1.25/24) [SIMULATED]
  [OK]       Default route: via 10.0.1.1 dev eth0 [SIMULATED]
  [OK]       Configured nameservers: 10.0.0.2 [SIMULATED]
  [CRITICAL] Resolve example.com: FAILED (timeout) [SIMULATED]
             ↳ Why:    Name could not be resolved: resolver unreachable, misconfigured, or record missing
             ↳ Next:   dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
  [CRITICAL] Nameserver 10.0.0.2 answers: no answer [SIMULATED]
             ↳ Why:    This resolver does not answer: down or UDP/TCP 53 blocked
             ↳ Next:   dig example.com @10.0.0.2  # check security group port 53
  [CRITICAL] DNS resolution: 0/2 names resolved [SIMULATED]
             ↳ Why:    No name resolves
             ↳ Next:   resolvectl status
  [OK]       TCP 1.1.1.1:443: open (12ms) [SIMULATED]
  [OK]       Outbound IP connectivity: 1 IP endpoint(s) reachable [SIMULATED]
  [INFO]     Listening TCP ports: 22 3306 [SIMULATED]
  [CRITICAL] HTTP port 80 listening: no [SIMULATED]
             ↳ Why:    Nothing listens on port 80: the web server is not running
             ↳ Next:   ss -tlnp | grep ':80'

  💡 Did you know? Linux load average also counts tasks waiting on disk I/O: high load can mean slow storage, not busy CPUs.

────────────────────────────────────────────────────────────────────────
  Disk & Memory Analysis
────────────────────────────────────────────────────────────────────────
  [OK]       Disk usage /: 54% (9.1 GiB free) [SIMULATED]
  [CRITICAL] Disk usage /var: 96% (812 MiB free of 20.0 GiB) [SIMULATED] (threshold: warn 80% / crit 90%)
             ↳ Why:    Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
             ↳ Next:   du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
  ↳ Largest directories in /var:
             │ /var/log=14.2G  /var/lib=3.9G  /var/cache=0.8G
             │ /var/log: nginx=11.8G journal=1.9G
  [OK]       Memory in use: 48% (2.0 GiB available of 3.8 GiB) [SIMULATED]
  [OK]       OOM killer events: 0 [SIMULATED]

  💡 Did you know? HTTP 418 'I'm a teapot' comes from RFC 2324, an April Fools' joke from 1998.

────────────────────────────────────────────────────────────────────────
  Performance Analysis
────────────────────────────────────────────────────────────────────────
  [OK]       Load per CPU core (5 min): 0.62 (load 2.71/2.48 on 4 cores) [SIMULATED]
  [OK]       CPU busy: 31% [SIMULATED]
  [OK]       CPU I/O wait: 2.1% [SIMULATED]
  [OK]       CPU steal (virtualisation): 0.4% [SIMULATED]
  [OK]       Pressure stall io (avg 60s): 1.8% of time [SIMULATED]
  [OK]       Zombie processes: 0 [SIMULATED]

  💡 Did you know? Ping is not proof of life: many firewalls drop ICMP while TCP services work fine.

────────────────────────────────────────────────────────────────────────
  Log Analysis
────────────────────────────────────────────────────────────────────────
  [WARNING]  Error lines in journal (since 24 hours ago): 10 [SIMULATED]
             ↳ Why:    Repeated errors grouped below
             ↳ Next:   journalctl -p err --since '24 hours ago'
  ↳ Most repeated messages (numbers normalised):
             │ 3 app: ERROR could not resolve api.payments.example: Temporary failure in name resolution
             │ 1 systemd: nginx.service: Main process exited, code=exited, status=<n>/FAILURE
             │ 1 systemd: nginx.service: Failed with result exit-code.
             │ 1 systemd: Failed to start A high performance web server.
             Error timeline (per hour):
             2026-10-19 09:00 ████████       2
             2026-10-19 10:00 ██████████████████████████████ 7
             2026-10-19 11:00 ████                   1
  ↳ Peak: 2026-10-19T10:00 (7 events)
  ↳ Evidence saved: evidence/log_errors.txt  (error lines (SIMULATED; note the redacted secrets))
  [WARNING]  Logs pattern 'unit_failed': 3 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
             ↳ Why:    A systemd unit failed to start or exited with an error
             ↳ Next:   systemctl --failed; journalctl -u <unit> -n 50
             ↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
             │ 2026-10-19T10:02:03+0200 demo-web-01 systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE
  [WARNING]  Logs pattern 'dns_fail': 3 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
             ↳ Why:    Name resolution failures: resolver unreachable/misconfigured or record missing
             ↳ Next:   cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
             ↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
             │ 2026-10-19T10:15:22+0200 demo-web-01 app[3120]: ERROR could not resolve api.payments.example: Temporary failure in name resolution
  [CRITICAL] Logs pattern 'disk_full': 2 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
             ↳ Why:    A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
             ↳ Next:   df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
             ↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
             │ 2026-10-19T09:58:12+0200 demo-web-01 nginx[2211]: 2026/10/19 09:58:12 [crit] 2211#0: pwrite() "/var/lib/nginx/body/0000000012" failed (28: No space le
  [WARNING]  Logs pattern 'timeout': 1 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
             ↳ Why:    Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
             ↳ Next:   # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
             ↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
             │ 2026-10-19T10:16:52+0200 demo-web-01 app[3120]: ERROR request failed password=[REDACTED] token=[REDACTED] (connection timed out)
  [WARNING]  Logs pattern 'auth_fail': 1 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
             ↳ Why:    Authentication failures: brute-force attempts or users with wrong credentials
             ↳ Next:   grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
             ↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
             │ 2026-10-19T11:01:09+0200 demo-web-01 sshd[4410]: Failed password for invalid user admin from 203.0.113.50 port 51122 ssh2

  💡 Did you know? Sysadmin haiku: It's not DNS. There's no way it's DNS. It was DNS.

────────────────────────────────────────────────────────────────────────
  Web Server Diagnostics
────────────────────────────────────────────────────────────────────────
  [CRITICAL] nginx service: failed [SIMULATED]
             ↳ Why:    The web server crashed or failed to start (config error, port conflict, full disk or OOM)
             ↳ Next:   systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
  [OK]       nginx -t (config test): syntax ok [SIMULATED]
  ↳ Config is valid: the start failure is not a syntax problem -> look at resources (disk)
  [CRITICAL] Port 443 listening: no [SIMULATED]
             ↳ Why:    A web server is installed but nothing listens on 443
             ↳ Next:   ss -tlnp | grep ':443'
  [CRITICAL] GET https://localhost/: connection failed (refused or unreachable) [SIMULATED]
             ↳ Why:    No HTTP answer
             ↳ Next:   curl -vk https://localhost/
  [WARNING]  TLS certificate shop.example.com:443: expires in 12 days (Oct 31 23:59:59 2026 GMT) [SIMULATED] (threshold: warn < 30d / crit < 7d)
             ↳ Why:    Certificate close to expiry: renewal may be failing
             ↳ Next:   certbot certificates; systemctl list-timers | grep -i certbot

  💡 Did you know? Tux the penguin was drawn by Larry Ewing in 1996 using GIMP.

────────────────────────────────────────────────────────────────────────
  Database Diagnostics
────────────────────────────────────────────────────────────────────────
  [OK]       postgresql@16-main service: active (running, enabled) [SIMULATED]
  [OK]       PostgreSQL accepting connections: yes [SIMULATED]
  [WARNING]  MySQL/MariaDB port 3306: listening on all interfaces (0.0.0.0) (suspected) [SIMULATED]
             ↳ Why:    Database reachable from the network: verify firewall/security group allow only app servers
             ↳ Next:   ufw status; security group rules  # check bind-address
  [OK]       Data dir /var/lib/postgresql filesystem: 54% used [SIMULATED]

  💡 Did you know? Linus Torvalds announced Linux on 25 Aug 1991 as 'just a hobby, won't be big and professional'.

────────────────────────────────────────────────────────────────────────
  Mail Server Diagnostics
────────────────────────────────────────────────────────────────────────
  [OK]       postfix service: active (running, enabled) [SIMULATED]
  [WARNING]  Mail queue: 142 message(s) [SIMULATED] (threshold: warn 50 / crit 500)
             ↳ Why:    Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
             ↳ Next:   postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
  [WARNING]  Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout (suspected) [SIMULATED]
             ↳ Why:    Outbound port 25 blocked: common default at cloud providers
             ↳ Next:   # check provider policy; use an authenticated relay on 587
             ↳ Limits: No email was sent during this test
  [WARNING]  Mail log pattern 'smtp_deferred': 1 occurrence(s) [SIMULATED] (threshold: KB rule; WARNING patterns need >= 1 hits)
             ↳ Why:    Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
             ↳ Next:   postqueue -p | tail  # check outbound TCP/25 and the 'said:' reply in the log
             ↳ Limits: Pattern match on log text: indicates a symptom, the cause must be confirmed
             │ Oct 19 10:20:01 demo-web-01 postfix/smtp[5120]: 4F2A1C: to=<a@example.org>, relay=none, delay=3600, status=deferred (connect to mx.example.org[192.0.2
  [OK]       MX example-shop.com: mx1.example-shop.com [SIMULATED]
  [WARNING]  DMARC example-shop.com: missing (suspected) [SIMULATED]
             ↳ Why:    No DMARC policy: large providers may reject
             ↳ Next:   dig TXT _dmarc.example-shop.com

  💡 Did you know? Linus Torvalds announced Linux on 25 Aug 1991 as 'just a hobby, won't be big and professional'.

────────────────────────────────────────────────────────────────────────
  Security & Access Checks
────────────────────────────────────────────────────────────────────────
  [OK]       SSH PermitRootLogin: prohibit-password [SIMULATED]
  [WARNING]  SSH PasswordAuthentication: yes (suspected) [SIMULATED]
             ↳ Why:    Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
             ↳ Next:   # prefer key-only auth; restrict port 22 to VPN/admin IPs
  [CRITICAL] Failed SSH logins (since 24 hours ago): 1843 [SIMULATED] (threshold: warn 20 / crit 500)
             ↳ Why:    Password-guessing against SSH
             ↳ Next:   # consider fail2ban; restrict port 22
             │     912 203.0.113.50
             │     455 198.51.100.23
             │     201 192.0.2.77
  [OK]       Firewall (ufw): active, 4 allow rules [SIMULATED]
  [WARNING]  Pending updates (apt): 37 total, 23 security [SIMULATED]
             ↳ Why:    Known vulnerabilities fixed upstream but not installed
             ↳ Next:   apt list --upgradable | grep security
  [OK]       Accounts with UID 0 besides root: none [SIMULATED]

  💡 Did you know? Tux the penguin was drawn by Larry Ewing in 1996 using GIMP.

────────────────────────────────────────────────────────────────────────
  Cloud & Container Checks
────────────────────────────────────────────────────────────────────────
  [INFO]     Platform: OpenStack (Nova) [SIMULATED]
  [OK]       Docker daemon: running (v27.3.1) [SIMULATED]
  [CRITICAL] Containers restarting: queue-worker [SIMULATED]
             ↳ Why:    Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
             ↳ Next:   docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
  ↳ Evidence saved: evidence/container_queue-worker.txt  (docker logs --tail 20 queue-worker (SIMULATED))
             │ 2026-10-19 10:31:02 ERROR cannot write /var/lib/worker/tmp/job-8812: No space left on device
             │ 2026-10-19 10:31:02 worker exited with code 1
  [OK]       Containers running: 11 [SIMULATED]
  [INFO]     Kubernetes: kubectl not configured on this host [SIMULATED]

────────────────────────────────────────────────────────────────────────
  Anomaly analysis & intervention priorities
────────────────────────────────────────────────────────────────────────
  Correlated findings (deterministic rules - correlation, not proof)

   P1  Full filesystem is the likely trigger of the service failures
        Evidence: Disk usage /var: 96% (812 MiB free of 20.0 GiB);nginx service: failed;log pattern 'No space left on device' present
        Meaning:  Logs contain 'No space left on device' and a filesystem is above the critical threshold while services are down. This is strong supporting evidence, but confirm in each unit's journal before acting.
        Next:     # Free space first (logs rotation, old releases, journalctl --vacuum-size after approval), then restart the affected services in a maintenance window.

   P1  DNS-layer failure: IP connectivity works, name resolution does not
        Evidence: Resolve example.com: FAILED (timeout);Nameserver 10.0.0.2 answers: no answer;DNS resolution: 0/2 names resolved;Outbound IP connectivity: 1 IP endpoint(s) reachable
        Meaning:  TCP connections to IP addresses succeed, so routing/firewall egress is fine; the fault is isolated to resolver configuration or the upstream DNS servers.
        Next:     cat /etc/resolv.conf; resolvectl status; dig example.com @<each-nameserver>  # check that UDP+TCP/53 to the resolver is allowed (security group/NACL)

   P1  nginx is down: closed HTTP ports and failing HTTP checks are consequences of the same fault
        Evidence: nginx service: failed;HTTP port 80 listening: no;Port 443 listening: no;GET https://localhost/: connection failed (refused or unreachable)
        Meaning:  The listener is missing because the process is not running. Fix the service start-up error first; the HTTP symptoms should then clear.
        Next:     nginx -t; journalctl -u nginx -n 50; ss -tlnp | grep -E ':80|:443'  # config test; port conflict?

   P2  SSH exposed to password guessing
        Evidence: Failed SSH logins (since 24 hours ago): 1843;SSH PasswordAuthentication: yes
        Meaning:  Many failed logins while password authentication is enabled increases the risk of credential compromise.
        Next:     # Restrict port 22 (security group/VPN), prefer key-only auth, consider fail2ban (changes require approval)

   P2  Deferred mail is consistent with blocked outbound SMTP (port 25)
        Evidence: Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout
        Meaning:  Many cloud providers block or throttle outbound TCP/25 by default; remote MX servers are then unreachable and mail stays deferred.
        Next:     # Verify provider policy, request unblock or use an authenticated relay on 587

  Anomalies by priority (25)

   P1  nginx service = failed  [web]
        Why:  The web server crashed or failed to start (config error, port conflict, full disk or OOM)
        Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
   P1  Resolve example.com = FAILED (timeout)  [network]
        Why:  Name could not be resolved: resolver unreachable, misconfigured, or record missing
        Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
   P1  Nameserver 10.0.0.2 answers = no answer  [network]
        Why:  This resolver does not answer: down or UDP/TCP 53 blocked
        Next: dig example.com @10.0.0.2  # check security group port 53
   P1  DNS resolution = 0/2 names resolved  [network]
        Why:  No name resolves
        Next: resolvectl status
   P1  HTTP port 80 listening = no  [network]
        Why:  Nothing listens on port 80: the web server is not running
        Next: ss -tlnp | grep ':80'
   P1  Disk usage /var = 96% (812 MiB free of 20.0 GiB)  [storage]
        Why:  Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
        Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
   P1  Logs pattern 'disk_full' = 2 occurrence(s)  [logs]
        Why:  A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
        Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
   P1  Port 443 listening = no  [web]
        Why:  A web server is installed but nothing listens on 443
        Next: ss -tlnp | grep ':443'
   P1  GET https://localhost/ = connection failed (refused or unreachable)  [web]
        Why:  No HTTP answer
        Next: curl -vk https://localhost/
   P1  Failed SSH logins (since 24 hours ago) = 1843  [security]
        Why:  Password-guessing against SSH
        Next: # consider fail2ban; restrict port 22
   P1  Containers restarting = queue-worker  [cloud]
        Why:  Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
        Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
   P2  System state = degraded  [system]
        Why:  systemd is up but at least one unit has failed
        Next: systemctl --failed
   P2  Error lines in journal (since 24 hours ago) = 10  [logs]
        Why:  Repeated errors grouped below
        Next: journalctl -p err --since '24 hours ago'
   P2  Logs pattern 'unit_failed' = 3 occurrence(s)  [logs]
        Why:  A systemd unit failed to start or exited with an error
        Next: systemctl --failed; journalctl -u <unit> -n 50
   P2  Logs pattern 'dns_fail' = 3 occurrence(s)  [logs]
        Why:  Name resolution failures: resolver unreachable/misconfigured or record missing
        Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
   P2  Logs pattern 'timeout' = 1 occurrence(s)  [logs]
        Why:  Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
        Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
   P2  Logs pattern 'auth_fail' = 1 occurrence(s)  [logs]
        Why:  Authentication failures: brute-force attempts or users with wrong credentials
        Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
   P2  TLS certificate shop.example.com:443 = expires in 12 days (Oct 31 23:59:59 2026 GMT)  [web]
        Why:  Certificate close to expiry: renewal may be failing
        Next: certbot certificates; systemctl list-timers | grep -i certbot
   P2  Mail queue = 142 message(s)  [mail]
        Why:  Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
        Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
   P2  Mail log pattern 'smtp_deferred' = 1 occurrence(s)  [mail]
        Why:  Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
        Next: postqueue -p | tail  # check outbound TCP/25 and the 'said:' reply in the log
   P2  Pending updates (apt) = 37 total, 23 security  [security]
        Why:  Known vulnerabilities fixed upstream but not installed
        Next: apt list --upgradable | grep security
   P3  MySQL/MariaDB port 3306 = listening on all interfaces (0.0.0.0) (suspected)  [database]
        Why:  Database reachable from the network: verify firewall/security group allow only app servers
        Next: ufw status; security group rules  # check bind-address
   P3  Outbound SMTP to gmail-smtp-in.l.google.com:25 = timeout (suspected)  [mail]
        Why:  Outbound port 25 blocked: common default at cloud providers
        Next: # check provider policy; use an authenticated relay on 587
   P3  DMARC example-shop.com = missing (suspected)  [mail]
        Why:  No DMARC policy: large providers may reject
        Next: dig TXT _dmarc.example-shop.com
   P3  SSH PasswordAuthentication = yes (suspected)  [security]
        Why:  Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
        Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs

────────────────────────────────────────────────────────────────────────
  Run summary
────────────────────────────────────────────────────────────────────────
   SIMULATED DATA - demo mode

  Host: demo-web-01 (SIMULATED)   Log window: since 24 hours ago

  OK 28   WARNING 14   CRITICAL 12   UNKNOWN 0   INFO 5

  Health score  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0/100
  Verdict       CRITICAL - immediate action required

  Report: ~/opscope/reports/20261020-215822_DEMO-incident/report.md

  Main menu
    1) Full System Audit
    2) Service Diagnostics  (systemd, failed units, kernel, NTP)
    3) Network & DNS        (routes, DNS, TCP, HTTP, ports)
    4) Disk & Memory        (filesystems, inodes, RAM, swap, OOM)
    5) Log Analysis         (grouped errors, timeline, explanations)
    6) Mail Server          (queue, ports, MX/SPF/DMARC, port 25)
    7) Database             (PostgreSQL, MySQL/MariaDB)
    8) Web Server           (nginx/Apache, HTTP, TLS, 5xx)
    9) Security & Access    (SSH, logins, firewall, updates)
   10) Performance          (load, CPU, I/O wait, steal, PSI)
   11) Cloud & Containers   (Docker, Kubernetes, platform)
   12) Generate Report
    q) Quit
  Choice: 12

────────────────────────────────────────────────────────────────────────
  Anomaly analysis & intervention priorities
────────────────────────────────────────────────────────────────────────
  Correlated findings (deterministic rules - correlation, not proof)

   P1  Full filesystem is the likely trigger of the service failures
        Evidence: Disk usage /var: 96% (812 MiB free of 20.0 GiB);nginx service: failed;log pattern 'No space left on device' present
        Meaning:  Logs contain 'No space left on device' and a filesystem is above the critical threshold while services are down. This is strong supporting evidence, but confirm in each unit's journal before acting.
        Next:     # Free space first (logs rotation, old releases, journalctl --vacuum-size after approval), then restart the affected services in a maintenance window.

   P1  DNS-layer failure: IP connectivity works, name resolution does not
        Evidence: Resolve example.com: FAILED (timeout);Nameserver 10.0.0.2 answers: no answer;DNS resolution: 0/2 names resolved;Outbound IP connectivity: 1 IP endpoint(s) reachable
        Meaning:  TCP connections to IP addresses succeed, so routing/firewall egress is fine; the fault is isolated to resolver configuration or the upstream DNS servers.
        Next:     cat /etc/resolv.conf; resolvectl status; dig example.com @<each-nameserver>  # check that UDP+TCP/53 to the resolver is allowed (security group/NACL)

   P1  nginx is down: closed HTTP ports and failing HTTP checks are consequences of the same fault
        Evidence: nginx service: failed;HTTP port 80 listening: no;Port 443 listening: no;GET https://localhost/: connection failed (refused or unreachable)
        Meaning:  The listener is missing because the process is not running. Fix the service start-up error first; the HTTP symptoms should then clear.
        Next:     nginx -t; journalctl -u nginx -n 50; ss -tlnp | grep -E ':80|:443'  # config test; port conflict?

   P2  SSH exposed to password guessing
        Evidence: Failed SSH logins (since 24 hours ago): 1843;SSH PasswordAuthentication: yes
        Meaning:  Many failed logins while password authentication is enabled increases the risk of credential compromise.
        Next:     # Restrict port 22 (security group/VPN), prefer key-only auth, consider fail2ban (changes require approval)

   P2  Deferred mail is consistent with blocked outbound SMTP (port 25)
        Evidence: Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout
        Meaning:  Many cloud providers block or throttle outbound TCP/25 by default; remote MX servers are then unreachable and mail stays deferred.
        Next:     # Verify provider policy, request unblock or use an authenticated relay on 587

  Anomalies by priority (25)

   P1  nginx service = failed  [web]
        Why:  The web server crashed or failed to start (config error, port conflict, full disk or OOM)
        Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
   P1  Resolve example.com = FAILED (timeout)  [network]
        Why:  Name could not be resolved: resolver unreachable, misconfigured, or record missing
        Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
   P1  Nameserver 10.0.0.2 answers = no answer  [network]
        Why:  This resolver does not answer: down or UDP/TCP 53 blocked
        Next: dig example.com @10.0.0.2  # check security group port 53
   P1  DNS resolution = 0/2 names resolved  [network]
        Why:  No name resolves
        Next: resolvectl status
   P1  HTTP port 80 listening = no  [network]
        Why:  Nothing listens on port 80: the web server is not running
        Next: ss -tlnp | grep ':80'
   P1  Disk usage /var = 96% (812 MiB free of 20.0 GiB)  [storage]
        Why:  Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
        Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
   P1  Logs pattern 'disk_full' = 2 occurrence(s)  [logs]
        Why:  A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
        Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
   P1  Port 443 listening = no  [web]
        Why:  A web server is installed but nothing listens on 443
        Next: ss -tlnp | grep ':443'
   P1  GET https://localhost/ = connection failed (refused or unreachable)  [web]
        Why:  No HTTP answer
        Next: curl -vk https://localhost/
   P1  Failed SSH logins (since 24 hours ago) = 1843  [security]
        Why:  Password-guessing against SSH
        Next: # consider fail2ban; restrict port 22
   P1  Containers restarting = queue-worker  [cloud]
        Why:  Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
        Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
   P2  System state = degraded  [system]
        Why:  systemd is up but at least one unit has failed
        Next: systemctl --failed
   P2  Error lines in journal (since 24 hours ago) = 10  [logs]
        Why:  Repeated errors grouped below
        Next: journalctl -p err --since '24 hours ago'
   P2  Logs pattern 'unit_failed' = 3 occurrence(s)  [logs]
        Why:  A systemd unit failed to start or exited with an error
        Next: systemctl --failed; journalctl -u <unit> -n 50
   P2  Logs pattern 'dns_fail' = 3 occurrence(s)  [logs]
        Why:  Name resolution failures: resolver unreachable/misconfigured or record missing
        Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
   P2  Logs pattern 'timeout' = 1 occurrence(s)  [logs]
        Why:  Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
        Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
   P2  Logs pattern 'auth_fail' = 1 occurrence(s)  [logs]
        Why:  Authentication failures: brute-force attempts or users with wrong credentials
        Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
   P2  TLS certificate shop.example.com:443 = expires in 12 days (Oct 31 23:59:59 2026 GMT)  [web]
        Why:  Certificate close to expiry: renewal may be failing
        Next: certbot certificates; systemctl list-timers | grep -i certbot
   P2  Mail queue = 142 message(s)  [mail]
        Why:  Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
        Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
   P2  Mail log pattern 'smtp_deferred' = 1 occurrence(s)  [mail]
        Why:  Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
        Next: postqueue -p | tail  # check outbound TCP/25 and the 'said:' reply in the log
   P2  Pending updates (apt) = 37 total, 23 security  [security]
        Why:  Known vulnerabilities fixed upstream but not installed
        Next: apt list --upgradable | grep security
   P3  MySQL/MariaDB port 3306 = listening on all interfaces (0.0.0.0) (suspected)  [database]
        Why:  Database reachable from the network: verify firewall/security group allow only app servers
        Next: ufw status; security group rules  # check bind-address
   P3  Outbound SMTP to gmail-smtp-in.l.google.com:25 = timeout (suspected)  [mail]
        Why:  Outbound port 25 blocked: common default at cloud providers
        Next: # check provider policy; use an authenticated relay on 587
   P3  DMARC example-shop.com = missing (suspected)  [mail]
        Why:  No DMARC policy: large providers may reject
        Next: dig TXT _dmarc.example-shop.com
   P3  SSH PasswordAuthentication = yes (suspected)  [security]
        Why:  Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
        Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs

────────────────────────────────────────────────────────────────────────
  Run summary
────────────────────────────────────────────────────────────────────────
   SIMULATED DATA - demo mode

  Host: demo-web-01 (SIMULATED)   Log window: since 24 hours ago

  OK 28   WARNING 14   CRITICAL 12   UNKNOWN 0   INFO 5

  Health score  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0/100
  Verdict       CRITICAL - immediate action required

  Report: ~/opscope/reports/20261020-215822_DEMO-incident/report.md

  Main menu
    1) Full System Audit
    2) Service Diagnostics  (systemd, failed units, kernel, NTP)
    3) Network & DNS        (routes, DNS, TCP, HTTP, ports)
    4) Disk & Memory        (filesystems, inodes, RAM, swap, OOM)
    5) Log Analysis         (grouped errors, timeline, explanations)
    6) Mail Server          (queue, ports, MX/SPF/DMARC, port 25)
    7) Database             (PostgreSQL, MySQL/MariaDB)
    8) Web Server           (nginx/Apache, HTTP, TLS, 5xx)
    9) Security & Access    (SSH, logins, firewall, updates)
   10) Performance          (load, CPU, I/O wait, steal, PSI)
   11) Cloud & Containers   (Docker, Kubernetes, platform)
   12) Generate Report
    q) Quit
  Choice: q

────────────────────────────────────────────────────────────────────────
  Anomaly analysis & intervention priorities
────────────────────────────────────────────────────────────────────────
  Correlated findings (deterministic rules - correlation, not proof)

   P1  Full filesystem is the likely trigger of the service failures
        Evidence: Disk usage /var: 96% (812 MiB free of 20.0 GiB);nginx service: failed;log pattern 'No space left on device' present
        Meaning:  Logs contain 'No space left on device' and a filesystem is above the critical threshold while services are down. This is strong supporting evidence, but confirm in each unit's journal before acting.
        Next:     # Free space first (logs rotation, old releases, journalctl --vacuum-size after approval), then restart the affected services in a maintenance window.

   P1  DNS-layer failure: IP connectivity works, name resolution does not
        Evidence: Resolve example.com: FAILED (timeout);Nameserver 10.0.0.2 answers: no answer;DNS resolution: 0/2 names resolved;Outbound IP connectivity: 1 IP endpoint(s) reachable
        Meaning:  TCP connections to IP addresses succeed, so routing/firewall egress is fine; the fault is isolated to resolver configuration or the upstream DNS servers.
        Next:     cat /etc/resolv.conf; resolvectl status; dig example.com @<each-nameserver>  # check that UDP+TCP/53 to the resolver is allowed (security group/NACL)

   P1  nginx is down: closed HTTP ports and failing HTTP checks are consequences of the same fault
        Evidence: nginx service: failed;HTTP port 80 listening: no;Port 443 listening: no;GET https://localhost/: connection failed (refused or unreachable)
        Meaning:  The listener is missing because the process is not running. Fix the service start-up error first; the HTTP symptoms should then clear.
        Next:     nginx -t; journalctl -u nginx -n 50; ss -tlnp | grep -E ':80|:443'  # config test; port conflict?

   P2  SSH exposed to password guessing
        Evidence: Failed SSH logins (since 24 hours ago): 1843;SSH PasswordAuthentication: yes
        Meaning:  Many failed logins while password authentication is enabled increases the risk of credential compromise.
        Next:     # Restrict port 22 (security group/VPN), prefer key-only auth, consider fail2ban (changes require approval)

   P2  Deferred mail is consistent with blocked outbound SMTP (port 25)
        Evidence: Outbound SMTP to gmail-smtp-in.l.google.com:25: timeout
        Meaning:  Many cloud providers block or throttle outbound TCP/25 by default; remote MX servers are then unreachable and mail stays deferred.
        Next:     # Verify provider policy, request unblock or use an authenticated relay on 587

  Anomalies by priority (25)

   P1  nginx service = failed  [web]
        Why:  The web server crashed or failed to start (config error, port conflict, full disk or OOM)
        Next: systemctl status nginx --no-pager -l; journalctl -u nginx -n 50 --no-pager
   P1  Resolve example.com = FAILED (timeout)  [network]
        Why:  Name could not be resolved: resolver unreachable, misconfigured, or record missing
        Next: dig example.com; dig example.com @1.1.1.1; cat /etc/resolv.conf
   P1  Nameserver 10.0.0.2 answers = no answer  [network]
        Why:  This resolver does not answer: down or UDP/TCP 53 blocked
        Next: dig example.com @10.0.0.2  # check security group port 53
   P1  DNS resolution = 0/2 names resolved  [network]
        Why:  No name resolves
        Next: resolvectl status
   P1  HTTP port 80 listening = no  [network]
        Why:  Nothing listens on port 80: the web server is not running
        Next: ss -tlnp | grep ':80'
   P1  Disk usage /var = 96% (812 MiB free of 20.0 GiB)  [storage]
        Why:  Filesystem filling up: logs not rotated, old backups, Docker images or deleted-but-open files
        Next: du -xh --max-depth=1 /var | sort -h | tail; journalctl --disk-usage; lsof +L1
   P1  Logs pattern 'disk_full' = 2 occurrence(s)  [logs]
        Why:  A filesystem is full: writes fail (logs, temp files, DB pages, sessions)
        Next: df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
   P1  Port 443 listening = no  [web]
        Why:  A web server is installed but nothing listens on 443
        Next: ss -tlnp | grep ':443'
   P1  GET https://localhost/ = connection failed (refused or unreachable)  [web]
        Why:  No HTTP answer
        Next: curl -vk https://localhost/
   P1  Failed SSH logins (since 24 hours ago) = 1843  [security]
        Why:  Password-guessing against SSH
        Next: # consider fail2ban; restrict port 22
   P1  Containers restarting = queue-worker  [cloud]
        Why:  Crash loop: the container exits and Docker restarts it (exit 137 = killed, often out of memory)
        Next: docker logs --tail 50 queue-worker; docker inspect -f '{{.State.OOMKilled}}' queue-worker
   P2  System state = degraded  [system]
        Why:  systemd is up but at least one unit has failed
        Next: systemctl --failed
   P2  Error lines in journal (since 24 hours ago) = 10  [logs]
        Why:  Repeated errors grouped below
        Next: journalctl -p err --since '24 hours ago'
   P2  Logs pattern 'unit_failed' = 3 occurrence(s)  [logs]
        Why:  A systemd unit failed to start or exited with an error
        Next: systemctl --failed; journalctl -u <unit> -n 50
   P2  Logs pattern 'dns_fail' = 3 occurrence(s)  [logs]
        Why:  Name resolution failures: resolver unreachable/misconfigured or record missing
        Next: cat /etc/resolv.conf; resolvectl status; dig <name> @<resolver>
   P2  Logs pattern 'timeout' = 1 occurrence(s)  [logs]
        Why:  Operations timing out: network latency/packet loss, firewall DROP, overloaded upstream or slow disk
        Next: # check latency (mtr), firewall rules, upstream load; correlate with CPU/IO pressure
   P2  Logs pattern 'auth_fail' = 1 occurrence(s)  [logs]
        Why:  Authentication failures: brute-force attempts or users with wrong credentials
        Next: grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
   P2  TLS certificate shop.example.com:443 = expires in 12 days (Oct 31 23:59:59 2026 GMT)  [web]
        Why:  Certificate close to expiry: renewal may be failing
        Next: certbot certificates; systemctl list-timers | grep -i certbot
   P2  Mail queue = 142 message(s)  [mail]
        Why:  Messages are not leaving: remote unreachable, port 25 blocked or DNS problem
        Next: postqueue -p | tail -30; grep 'status=deferred' /var/log/mail.log | tail
   P2  Mail log pattern 'smtp_deferred' = 1 occurrence(s)  [mail]
        Why:  Delivery temporarily failing (remote unreachable, greylisting, port 25 blocked, TLS issues)
        Next: postqueue -p | tail  # check outbound TCP/25 and the 'said:' reply in the log
   P2  Pending updates (apt) = 37 total, 23 security  [security]
        Why:  Known vulnerabilities fixed upstream but not installed
        Next: apt list --upgradable | grep security
   P3  MySQL/MariaDB port 3306 = listening on all interfaces (0.0.0.0) (suspected)  [database]
        Why:  Database reachable from the network: verify firewall/security group allow only app servers
        Next: ufw status; security group rules  # check bind-address
   P3  Outbound SMTP to gmail-smtp-in.l.google.com:25 = timeout (suspected)  [mail]
        Why:  Outbound port 25 blocked: common default at cloud providers
        Next: # check provider policy; use an authenticated relay on 587
   P3  DMARC example-shop.com = missing (suspected)  [mail]
        Why:  No DMARC policy: large providers may reject
        Next: dig TXT _dmarc.example-shop.com
   P3  SSH PasswordAuthentication = yes (suspected)  [security]
        Why:  Password logins allowed: acceptable only with strong passwords + fail2ban / restricted source IPs
        Next: # prefer key-only auth; restrict port 22 to VPN/admin IPs

────────────────────────────────────────────────────────────────────────
  Run summary
────────────────────────────────────────────────────────────────────────
   SIMULATED DATA - demo mode

  Host: demo-web-01 (SIMULATED)   Log window: since 24 hours ago

  OK 28   WARNING 14   CRITICAL 12   UNKNOWN 0   INFO 5

  Health score  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0/100
  Verdict       CRITICAL - immediate action required

  Report: ~/opscope/reports/20261020-215822_DEMO-incident/report.md

← Back to the lab