Monitor a remote computer securely using SNMP (OpenBSD)

The Simple Network Management Protocol (SNMP) lets you query a computer for system monitoring info, like uptime and hard drive capacity

OpenBSD has an SNMP server built in (snmpd), as well as an SNMP client (snmpctl)

OpenBSD includes SNMP info ("mibs") for CARP, relayd, and pf as well as the standard ones

The OpenBSD client is limited to version 2 of SNMP, so we'll use net-snmp, which supports version 3 and therefore authentication and encryption. There's a package for that.

Server (10.0.2.35)

# cat /etc/snmpd.conf

listen on 10.0.2.35
seclevel enc
user "snmp" authkey "s3cr3t00" enckey "s3cr3t00" enc aes auth hmac-sha1

# chown root:_snmpd /etc/snmpd.conf `# root is owner, _snmpd is group`
# chmod u=w,g=r,o= /etc/snmpd.conf `# root write, _snmpd read, other nothing`
# rcctl enable snmpd
# rcctl start snmpd

Need to check your config?

# snmpd -n

Need to debug snmpd?

# snmpd -dvv

Client

# pkg_add net-snmp
# cat /etc/snmp/snmp.conf

defSecurityName snmp
defSecurityLevel authPriv
defPassphrase "s3cr3t00"
defAuthType SHA
defPrivType AES

# chown root:wheel `# root is owner, wheel is group`
# chmod u=w,g=r,o= /etc/snmp/snmp.conf `# root write, wheel read, other nothing`
# snmpwalk 10.0.2.35 | less `# shows much info from remote system`

Forward syslogs to a central server using TLS (OpenBSD)

Syslogd runs by default on OpenBSD and is used to collect log messages on a host

You can see (some of) the logs using

$ tail /var/log/messages

And you can add a message using

$ logger 'hello'

Say you have dozens or hundreds of servers. You don't want to login to each one and look at each log. Instead, it makes sense to forward them to a central server. It's also safer, as it makes it more difficult for a hacker to hide his tracks.

We don't want to send potentially sensitive logs in plain text, so we'll use TLS

Server (10.0.2.35 aka 'loghost')

# cd /etc/ssl/private
# openssl genrsa -out loghost.key 2048 `# generate 2048-bit RSA key`
# openssl req -x509 -new -key loghost.key -out ../loghost.crt -days 365 `# generate cert. set common name to loghost`
# echo '10.0.2.35 loghost' >> /etc/hosts
# rcctl set syslogd flags "-S loghost" `# -S means use TLS`
# rcctl restart syslogd
# tail -f /var/log/messages `# watch for syslog messages`

Client (10.0.2.36 aka 'logclient')

Add to /etc/syslog.conf:

*.notice;auth,authpriv,cron,ftp,kern,lpr,mail,user.none  @tls://loghost
auth,daemon,syslog,user.info;authpriv.kern.debug  @tls://loghost

# echo '10.0.2.35 loghost' >> /etc/hosts
# scp nonRootSSHUser@loghost:/etc/ssl/loghost.crt /etc/ssl/trusted.crt `# copy the cert from server`
# rcctl set syslogd flags "-h -C /etc/ssl/trusted.crt" `# use hostnames instead of IPs and set the TLS trust store`
# rcctl restart syslogd
# logger hello from logclient `# you should see this on loghost`

If you need to debug syslogd, run /usr/sbin/syslogd -d

Setting up OpenBSD's LDAP Server (ldapd) with StartTLS and SASL

OpenBSD has its own LDAP server, ldapd. Here's how to configure it for use with StartTLS and SASL authentication

I don't cover common LDAP operations, like adding users, as that's covered well elsewhere

Server (ldapserver / 10.0.2.35)

1. Create a self-signed TLS certificate

# cd /etc/ldap/certs
# openssl genrsa -out ldapserver.key 2048
# openssl req -x509 -new -key ldapserver.key -out ldapserver.crt -days 365
# `# enter ldapserver as the canonical name`

Configure ldapd:

# touch /etc/ldapd.conf
# chmod o-r /etc/ldapd.conf 
# cat /etc/ldapd.conf

schema "/etc/ldap/core.schema"
schema "/etc/ldap/inetorgperson.schema"
schema "/etc/ldap/nis.schema"

listen on lo0 tls certificate ldapserver  
listen on em0 tls certificate ldapserver
listen on "/var/run/ldapi" 

Check your configuration:

# /usr/sbin/ldapd -n
configuration ok

Enable and start ldapd:

# rcctl enable ldapd
# rcctl start ldapd
ldapd(ok)

Copy the cert to /tmp so we can scp it as not root from the client:

cp /etc/ldap/certs/ldapserver.crt /tmp

Need to debug ldapd?

# /usr/sbin/ldapd -d -vv
# ldapctl log verbose

Client

Trust the server's certificate:

# echo '10.0.2.35    ldapserver' >> /etc/hosts
# scp notRoot@ldapserver:/tmp/ldapserver.crt >> /etc/ssl/trusted.crt

Install openldap-client (this installs cyrus-sasl as well):

# pkg_add openldap-client

Configure it:

# cat /etc/openldap/ldap.conf
TLS_CACERT    /etc/ssl/trusted.crt

Start saslauthd

# rcctl enable saslauthd
# rcctl start saslauthd
saslauthd(ok)

Connect to ldapd (-ZZ means force TLS, use -H to specify URI):

# ldapsearch -H ldap://ldapserver -ZZ
SASL/PLAIN authentication started
Please enter your password:

Need to debug saslauthd?

# /usr/local/sbin/saslauthd -a getpwent  -d

Need to debug OpenLDAP client?

# ldapsearch -d1 -v -ZZ
# ldapadd -d1 -v -ZZ

Crush Tools convdate

Convert dates on the command line

$ echo '7-25-2015-12:12:12' | convdate --verbose
2016-07-25-12:12:12

$ echo '7/25/16' | convdate --verbose -i '%m/%d/%y' -o '%Y-%m-%d'
2016-07-25

where
i is the input format, as per http://pubs.opengroup.org/onlinepubs/009695399/functions/strptime.html 
o is the output format, as per http://pubs.opengroup.org/onlinepubs/009695399/functions/strftime.html

Crush Tools subtotal

Given foo.csv:

group1,1,2
group1,1,3
group1,1,4
group2,1,5
group2,1,6

We want to subtotal columns 2 and 3 (individually) by group in column1:

subtotal --key 1 --sum 2,3 --delim , -i foo.csv

group1,1,2
group1,1,3
group1,1,4
,3,9

group2,1,5
group2,1,6
,2,11

Easy XPath against HTML

UPDATE. Even easier with xidel:

xidel 'http://example.com' --extract '//title'

ORIGINAL:

curl -L example.com | \
  tidy -asxml -numeric -utf8 | \
  sed -e 's/ xmlns.*=".*"//g' | \
  xml select -t -v "//title" -n

1. Get the HTML content from http://example.com using curl
2. Use HTML Tidy to tidy it up, covert it to XHTML, change &entities; to numeric ones, and set the encoding as UTF-8
3. Use sed to remove the XML namespace, for simpler XPaths
4. Use XML Starlet to select by XPath

You can output multiple columns like so:

curl -L example.com | \
  tidy -asxml -numeric -utf8 | \
  sed -e 's/ xmlns.*=".*"//g' | \
  xml select -t -v "//title" -o ','-v "//another" -n

Crush Tools dbstream with PostgreSQL

There's a command line library from Google called Crush Tools. It hasn't been updated for a while, but a few commands look useful, like pivot and aggregate.

There's also one called dbstream, which is a perl script to stream to/from a database.

Here's how to run it (on OS X anyways):

Install it (crush-tools, Perl package manager, Perl PostgreSQL driver):

$ brew install crush-tools
$ curl -L https://cpanmin.us | perl - --sudo App::cpanminus
$ sudo cpanm DBD::Pg

And run it:

$ dbstream -de , -ds dbi:Pg:dbname=foo -u alice -p secret -s "select 1,2;"

or insert each record from a file:

$ cut -d, -f1-2 input.csv | dbstream ... -s "insert into tbl values (?,?);"

or look up each record from a file:

$ cut -d, -f1 input.csv | dbstream ... -s "select name from tbl where code=?;"

or update up each record from a file:

$ awk -F, '{print $2","$1}' input.csv | dbstream ... -s "update tbl set name=? where id=?;"

where

-de is the input & output field separator
-ds is a Perl DBI datasource
-u username
-p password
-s SQL

It's here if it's not in your package manager: https://github.com/google/crush-tools