TIL: Geany and Shellcheck

I've been using Geany for a while. When I selected it, I was looking for a cross-platform text editor with convenience features for writing code because all of the IDEs I had tried were cumbersome. I wanted syntax highlighting and code folding. I was happy that Geany also includes auto-completion, project support, vertical selection (when that's useful, nothing else will work as quickly), and plugin extensibility.

More recently, I ran across Shellcheck and was impressed with what it detects. Of course I could use it online or on the command line, but wouldn't it be nice to have it integrated with Geany also? Sure it would! So I proceeded to look for a Shellcheck plugin for Geany, in a case of retrospectively-humorous linear thinking.

The punchline is that Geany doesn't need a plugin; it has a far simpler approach. When I open one of my bash scripts in Geany, the first menu item under the Build menu is Lint. Guess what lint on a shell script runs? Of course it runs shellcheck! So today I learned that the feature I wanted in Geany was already there in the most appropriate place ... once I broke out of the linear thinking.

Virsh

We all know how fun brain dumps are to read, so move along, nothing to see here.

Today was my second day arguing with learning virsh. Here's what I've learned so far (with some bruises).

Basic virsh

reference

You'll need to install various tools, but that should be readily search-able. I had the tools and some existing VMs as my starting point, but I had a lot to learn.

Create (build) a new VM

• define (export) NEWVM, OLDVM, HOST, DOMAIN, and ME (assuming your VG, LV, and VM naming scheme matches mine; otherwise, watch out!)
• sudo lvcreate -n vm_${NEWVM} -L 8192M /dev/vg_${HOST} # from http://virt-tools.org/learning/install-with-command-line/
• The rest of this is mostly http://manpages.ubuntu.com/manpages/trusty/man1/virt-builder.1.html
• virt-builder --list
• virt-builder --notes ubuntu-14.04
• From that, I learned that I want to add --firstboot-command "dpkg-reconfigure openssh-server" to virt-builder.
• Create a file with your desired initial root password. Mine was /tmp/password.
• virt-builder ubuntu-14.04 -o /dev/vg_${HOST}/lv_${NEWVM} --firstboot-command "dpkg-reconfigure openssh-server" --root-password file:/tmp/passwd --hostname ${NEWVM}.${DOMAIN}
• Now I leverage that I already had some working VMs of the same OS type and version:
• virsh dumpxml ${OLDVM} | sed 's/${OLDVM}/${NEWVM}/g' | grep -v "mac address\|uuid" > ${NEWVM}.xml
• virsh define ${NEWVM}.xml # persistent VM
• virsh start ${NEWVM}
• Alternatively, 'virsh create ${NEWVM}.xml' is a transient VM that goes away when shutdown
• Very awesomely, console access was available by default! If it weren't, you'll want to configure it.
• virsh console ${NEWVM} # log in as root with the password set in the file
• useradd -G sudo -s /bin/bash -d /home/${ME} -m -c "Your Name" ${ME}
• passwd ${ME} # set your password
• If console doesn't work, you'll need to use some brute force.
  • virsh domiflist ${NEWVM}
  • ipv6calc --action prefixmac2ipv6 --in prefix+mac --out ipv6addr fe80:: [that MAC]
  • ssh -l root [that IP]
• Add DNS records for the new VM.
• When you're done, virsh shutdown ${NEWVM}

Clone an Ubuntu VM

• Read https://help.ubuntu.com/community/KVM/Virsh.
• virsh list --all
• virsh shutdown ${OLDVM}
• virsh list --all # verify that shutdown completed
• sudo virt-clone --original ${OLDVM} --name ${NEWVM} --prompt # first time, or
• sudo virt-clone --original ${OLDVM} --name ${NEWVM} -m [previous MAC] --prompt # subsequently, if you're learning by breaking as I did
  • answer /dev/vg_${HOST}/vm_${NEWVM} to prompt if you named yours like mine
• virsh list --all
• virsh start ${OLDVM}
• sudo virt-sysprep --hostname ${NEWVM}.${DOMAIN} --enable cron-spool,dhcp-client-state,dhcp-server-state,logfiles,mail-spool,random-seed,ssh-hostkeys,yum-uuid -d ${NEWVM}
• virsh start ${NEWVM}
• The clone has my user account and other niceties, but I don't know its address. Luckily I can use available information and link-local IPv6.
• virsh domiflist ${NEWVM}
• ipv6calc --action prefixmac2ipv6 --in prefix+mac --out ipv6addr fe80:: [that MAC]
• ssh [replace with that IPv6 address]%br0 # replace with your local network interface after '%'
  • sudo dpkg-reconfigure openssh-server # if you get ssh errors and have to use console
  • grep "${OLDVM}\|${NEWVM}" /etc/hostname # always verify!!!
  • sudo sed -i 's/${OLDVM}/${NEWVM}/g' /etc/hostname
  • grep "${OLDVM}\|${NEWVM}" /etc/hostname # and check my work!
• virsh reboot ${NEWVM}
• Add DNS records for the new VM.
• virsh shutdown ${NEWVM} # when you're done

Clone a CentOS VM

• Read https://help.ubuntu.com/community/KVM/Virsh. This was a CentOS 7 clone.
• virsh list --all
• virsh shutdown ${OLDVM}
• virsh list --all # verify that shutdown completed
• sudo virt-clone --original ${OLDVM} --name ${NEWVM} --prompt # first time, or
• sudo virt-clone --original ${OLDVM} --name ${NEWVM} -m [previous MAC] --prompt # subsequently, if you're learning by breaking as I did
  • answer /dev/vg_${HOST}/vm_${NEWVM} to prompt if you named yours like mine
• virsh list --all
• virsh start ${OLDVM}
• sudo virt-sysprep --hostname ${NEWVM}.${DOMAIN} --enable cron-spool,dhcp-client-state,dhcp-server-state,logfiles,mail-spool,random-seed,ssh-hostkeys,yum-uuid -d ${NEWVM}
• virsh start ${NEWVM}
• The clone has my user account and other niceties, but I don't know its address. Luckily I can use available information and link-local IPv6.
• virsh domiflist ${NEWVM}
• ipv6calc --action prefixmac2ipv6 --in prefix+mac --out ipv6addr fe80:: [that MAC]
• ssh [replace with that IPv6 address]%br0 # replace with your local network interface after '%'
  • grep "${OLDVM}\|${NEWVM}" /etc/hostname # always verify!!!
  • sudo sed -i 's/${OLDVM}/${NEWVM}/g' /etc/hostname
  • grep "${OLDVM}\|${NEWVM}" /etc/hostname # and check my work!
  • sudo vgrename centos_${OLDVM} centos_${NEWVM} # or use lvm, see below
  • grep "${OLDVM}\|${NEWVM}" /etc/fstab
  • sudo sed -i 's/${OLDVM}/${NEWVM}/g' /etc/fstab # only needed if vgrename used
  • grep "${OLDVM}\|${NEWVM}" /etc/fstab
  • grep "${OLDVM}\|${NEWVM}" /etc/default/grub
  • sudo sed -i 's/${OLDVM}/${NEWVM}/g' /etc/default/grub # only needed if vgrename used
  • Now is a good time to add the RHEL7 console tips below!
  • grep "${OLDVM}\|${NEWVM}" /etc/default/grub
  • sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  • grep "${OLDVM}\|${NEWVM}" /boot/grub2/grub.cfg
  • exit # log out of ${NEWVM} now
• virsh reboot ${NEWVM}
• Add DNS records for the new VM.
• virsh shutdown ${NEWVM} # when you're done

Enable CentOS Console Access

Typically, you don't care about console access until it's the only way to get out of trouble. So enable it right away if it isn't working yet. Check with 'virsh console ${NEWVM}' and press return at least once to see if you get a login prompt.

The directions for RHEL6 are https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Troubleshooting_-Troubleshooting_with_serial_consoles.html. Edit the /boot/grub/grub.conf file to append this to the kernel line: console=tty0 console=ttyS0,115200

The directions for RHEL7 are http://www.daemonzone.net/e/17/, to add the following lines to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

Since mine already had GRUB_CMDLINE_LINUX (to apply it to all the menu entries), I just appended " console=tty0 console=ttyS0,115200n8" to that line inside the quotes, then added the next two lines. Next you need to rebuild the grub.cfg file:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

LVM

[${ME}@${NEWVM} ~]$ sudo lvm
lvm> pvdisplay
lvm> vgrename centos_${OLDVM} centos_${NEWVM}
  Volume group "centos_${OLDVM}" successfully renamed to "centos_${NEWVM}"
lvm> pvdisplay
lvm> lvdisplay
lvm> exit

Loose Ends

sudo virt-inspector -d ${NEWVM}
sudo virt-filesystems -d ${NEWVM}
sudo virt-df -d ${NEWVM}
sudo virt-df
sudo virt-edit -d ${BrokenGuest} /boot/grub2/grub.cfg
sudo virt-rescue -d ${BrokenGuest}

CLI to Google Calendar

After a few weeks of missing my old cli access to Google Calendar, and pushing the Try It! button with no success (yes, I customized it, but no, what I tried didn't work for me), I went looking (again) for another tool.

Hurrah! Look to gcalcli!

First I tried "pip install gcalcli; pip install vobject parsedatetime" but "which gcalcli" came up empty. I read the log file mentioned at the end of the pip install, ran a search on the most obvious error message, and re-ran it with sudo. Now it's working! I recommend "gcalcli list" to start.

The command I expect to use most often is

gcalcli --calendar 'Calendar' --title 'what is it' --where there --when '12/19/2014 10:00 PM' --duration 60 --description 'yay for description field finally!' add --details url

, and next

gcalcli --calendar 'Calendar' quick 'quick add text' --details url

(possibly). I have always wanted to be able to populate the description from the command line, but APIv2 only supported what's in Quick Add. Now that I've got another easy tool, I can forgive APIv3 for moving my cheese since it supports a feature I wanted.

Now if you'll excuse me, I need to pound on this new command line tool ...

Update: Oh, it works so well! I start at Spot the Station, I remove the irrelevant lines (too early, too short, or too low) with PrintWhatYouLike, and then I copy the remaining (relevant) lines.

That then becomes the starting point to fun with regular expressions! Although pbpaste is a Mac OS X tool hailing back to the NeXT days, Linux has tools like xsel and xclip that I hope are roughly equivalent.

clear; pbpaste | awk -F $'\t' '{ print "gcalcli --calendar \"Calendar Name\" --title \"ISS sighting\" --where \"Durham, NC\" --reminder 20 --when \""$1"\" --duration "$2" --description \""$3" max elevation, "$4" approach, "$5" depart\" add --details url" }' | sed 's/\ min\ /\ /g;s/\ above/°\ above/g'; echo

I like to glance at the commands before running them, just to be sure I haven't made some hideous formatting error about to be replicated at the speed of automation, and I like to check at least one of the resulting URIs to be sure the calendar entry survived the transfer. So far, so great!

S5 to PDF

I would rather use S5 for presentations than a proprietary solution. (I think Landslide is cool, and it has the feature to append #number to the URI to jump to that slide, but I can integrate S5 into my PmWiki so I stick with S5.) After a recent conference presentation, I was asked for the PDF version of my slides, and I was stumped for a moment. Then I found this post that turned me on to PrinceXML to work with S5. I thought about explaining how much I like Prince (after one use), but Ryan Tomayko already wrote that PrinceXML is extremely impressive. So I'll just say that I didn't like the initial output, and the documentation came to my rescue! I like well documented software that does what I need to do.

The actual command I used was prince --media projection -s page.css input.html -o output.pdf with the following for page.css:

@page { size: 1600px 900px; margin: 0 0 0 0; }

Wunderlist

So I have been experiencing the Wunderlist "Synchronizing failed" error, and finally decided it wasn't going away on its own so I should do something about it. Since I'm still using WL1, I wanted NOT to logout. Luckily, I can extract my data, using Wunderlist-Helpers, or even simpler using the command line.

Mac OS X has come with sqlite3 for a while now, so no installation needed.

$ find ~ -name "wunderlist.db" -print 2>/dev/null
~/Library/Application Support/Titanium/appdata/com.wunderkinder.wunderlist/wunderlist.db

(be sure to escape the space with a backslash!!!)

$ cd ~/Library/Application\ Support/Titanium/appdata/com.wunderkinder.wunderlist/
$ sqlite3 -csv -header wunderlist.db "select * from tasks left join lists on tasks.list_id=lists.id;" > ~/Desktop/wunderlist_export.csv
$ open ~/Desktop
$ open ~/Desktop/wunderlist_export.csv

Now you should see your desktop folder with the newly created export file, and it should be open in your default CSV viewer!

I think I'm going to go from Wunderlist back to Google Tasks.

IPv6 and Solaris 9

Since static IPv6 on Solaris 9 wasn't fun enough on its own, it turns out that my Solaris 9 servers start losing a lot of IPv6 packets after a reboot. The problem appears to be in.ndpd (what should be the IPv6 neighbor discovery protocol daemon, but appears primarily to be IPv6 routing instead), and syslog reports in.ndpd[PID]: [ID 302683 daemon.error] router_add_k: RTM_ADD (interface bge0): Network is unreachable RTM_ADD every 1 to 5 minutes.

The fix is to shut down IPv6 (it's not working anyway, so go ahead) and then restore IPv6; the only difference is that in.ndpd isn't running. However, just TERMinating in.ndpd isn't enough for IPv6 traffic to flow freely. Try this instead:

# kill in.ndpd

ps -ef | grep ndp

sudo kill -TERM 11292

# bring down the IPv6 interfaces

sudo ifconfig -a6

sudo ifconfig bge0:1 inet6 down unplumb

sudo ifconfig bge0 inet6 down unplumb

sudo ifconfig lo0 inet6 down unplumb

# flush IPv6 routes

sudo route flush -inet6

# verify that IPv6 is gone

sudo ifconfig -a6

netstat -nr

# restore IPv6 interfaces

sudo ifconfig lo0 inet6 plumb up

sudo ifconfig bge0 inet6 plumb up

sudo ifconfig bge0:1 inet6 plumb up your:ipv6:address::here/64

# verify interfaces

sudo ifconfig -a6

sudo ifconfig -a

# restore default IPv6 route

sudo route add -inet6 default your:ipv6:default:router::here

# verify route table

netstat -nr

I'm trying to stay off my diatribe soap box here, but I'm happy that Solaris 9 will be replaced with RHEL 6 in the coming months.

Static IPv6 on Solaris 10

I tried to avoid it since I'm retiring these Solaris boxes soon, but I had to put static IPv6 on Solaris 10 so I could troubleshoot. So here's how to do it, and it's not quite the same as Solaris 9.

First, get your baseline.

sudo ifconfig -a6

sudo ifconfig -a

netstat -nr

Start with loopback, the easy one.

cd /etc

sudo touch hostname6.lo0

sudo ifconfig lo0 inet6 plumb up

Move on to an actual interface, changing eri0 to your interface as needed. This is a change from the Solaris 9 approach!

sudo vi hostname6.eri0

This time, the /etc/hostname6.eri0 file should contain 'addif ipv6:addr:ess/64 up' – not just the IPv6 address/prefix, but the extra directives as well.

Now make it so!

sudo ifconfig eri0 inet6 plumb up

sudo ifconfig eri0:1 inet6 plumb

sudo ifconfig eri0:1 inet6 ipv6:host:addy/64 up

Add a route to the default router, and put its address in that file to survive reboots.

sudo route add -inet6 default ipv6:rtr:addy

sudo vi /etc/defaultrouter6

You'll want ndp running too.

sudo /usr/lib/inet/in.ndpd -a

Check your work against your baseline.

sudo ifconfig -a6

sudo ifconfig -a

netstat -nr

OK! That seems to work for now.

static IPv6 on Solaris 9

We use static IP addresses on our name servers, and I wanted IPv6 support on our public (and publicized) DNS too.

Doing this on Linux is straightforward and very similar to IPv4, as it should be, and you shouldn't have any trouble finding that information.

But IPv6 on Solaris, let alone Solaris 9, let alone static instead of SLAAC? Not as easy to find or do. Argh! I wasted a whole day on this, although I suppose it wasn't a wasted day since it accelerated my plan to eradicate SPARC and Solaris. (Uniformity is easier to manage.) So here's how you do it.

First, check for any existing IPv6.

sudo ifconfig -a6

While we're here, see what we've got (baseline).

sudo ifconfig -a

Since we'll need to add an IPv6 gateway, baseline the routing table too.

sudo netstat -nr

OK, baseline data collection done!

Start with an easy win, adding reboot-safe ::1 IPv6 localhost.

sudo touch /etc/hostname6.lo0

Bring up ::1. Don't forget, Solaris has that concept of plumbing the interface before it will come up.

sudo ifconfig lo0 inet6 plumb up

Depending on your interface name(s), replace bge0 with the correct interface for your system. First, tell the interface to support IPv6.

sudo ifconfig bge0 inet6 plumb up

Now you have to create a virtual interface for the IPv6 address (and yes, the address types co-exist on RHEL and Mac OS X) and bring it up. Since this is a static IPv6 address for a server, drop it in here.

sudo ifconfig bge0:1 inet6 plumb up your:ipv6:address:here

Now make it reboot-safe by putting the IPv6 address in the file /etc/hostaname6.interface.

sudo vi /etc/hostname6.bge0 # add the IPv6 addr

Not quite done! If this is a routed address, the Solaris box will need an IPv6 gateway too.

sudo route add -inet6 default your:ipv6:gateway:address

Make that reboot-safe too, by adding just the IPv6 router interface to /etc/defaultrouter6 too.

Make sure the updates look correct.

sudo ifconfig -a6

sudo ifconfig -a

netstat -nr

If you can ping6 your IPv6 gateway and IPv6 hosts outside your VLAN, it works! And it's not hard once you know what to do.

Just for reference, here's SLAAC IPv6 for Solaris 9 too; update with the right interface name from ifconfig -a as needed. Do the same baseline checks first! Then ...

sudo cp /etc/hostname.eri0 /etc/hostname6.eri0

sudo ifconfig eri0 inet6 plumb up

The RA should take care of IPv6 address prefix and routing for you.

From the depths of FreeRADIUS

Two pieces of advice on FreeRADIUS.

1. Keep up with new versions. There are improvements in each version, so don't just copy over the same config files: make those changes in the new files so you keep up with the times.
2. If you're starting with a fresh install, don't jump in and make changes right away. Test the freshly installed server first to be sure it works. Make a change, then restart the server to be sure it still works. Another change, another restart, another test. Debug mode -XXX is your friend.

And I finally eliminated a FreeRADIUS warning I saw while in debug mode. This warning is very common now because FreeRADIUS has been upgraded internally but not all of the sample files have been tidied up to reflect the improvement (see my recommendations above!). In debug mode, you might see Info: [ldap] WARNING: Deprecated conditional expansion ":-". See "man unlang" for details and man unlang wasn't particularly enlightening on first pass since it wasn't obvious until I read this post by Mr. FreeRADIUS himself where he explains why this change is A Good Thing. The key is to use more percents and more braces. The line that causes the error for ldap is in the modules/ldap file:

filter = "(uid=%{Stripped-User-Name:-%{User-Name}})"

It needs to updated like so:

filter = "(uid=%{%{Stripped-User-Name}:-%{User-Name}})"

This worked to eliminate the warning before I ditched LDAP because it wasn't matching what I needed.

rsync script

I wrote a quick bash script about a month ago to do my daily rsync, but I had a problem that took the quick and easy fun out of the script for several days. What worked on the command line to handle spaces didn't work in the script! The answer was buried in the rsync FAQ:

The [rsync] command line is interpreted by the remote shell and thus the spaces need to arrive on the remote system escaped so that the shell doesn't split such filenames into multiple arguments.

That's the core of the problem, but backslashes didn't seem to help. I went for the ? solution, to match one character that happens to be a space, instead.

Today I discovered that I probably could have stayed with what worked on the command line (but not in the script) if I had used eval on the scripted rsync command.

Latest version of rsync

I started with these directions to compile rsync, but updated to the latest version, 3.0.6, with my own variants to log every step noted in red.

Compile rsync 3.0.6

Follow these instructions in Terminal on both the client and server to download and compile rsync 3.0.6:

Download and unarchive rsync and its patches

cd ~/Desktop

curl -O http://rsync.samba.org/ftp/rsync/rsync-3.0.6.tar.gz

tar -xzvf rsync-3.0.6.tar.gz

rm rsync-3.0.6.tar.gz

curl -O http://rsync.samba.org/ftp/rsync/rsync-patches-3.0.6.tar.gz

tar -xzvf rsync-patches-3.0.6.tar.gz

rm rsync-patches-3.0.6.tar.gz

cd rsync-3.0.6

Apply patches relevant to preserving Mac OS X metadata

patch -p1 <patches/fileflags.diff

patch -p1 <patches/crtimes.diff

Configure, make, install

./prepare-source 2>&1 | tee prepare-source.out

./configure 2>&1 | tee configure.out

make 2>&1 | tee make.out

sudo make install 2>&1 | tee make-install.out

Verify your installation

[hope:~] /usr/local/bin/rsync --version

rsync version 3.0.6 protocol version 30

Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.

Web site: http://rsync.samba.org/

Capabilities:

64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,

socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,

append, ACLs, xattrs, iconv, symtimes, file-flags

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you

are welcome to redistribute it under certain conditions. See the GNU

General Public Licence for details.

By default, rsync will be installed in /usr/local/bin. If that isn't in your path, you will need to call your new version of rsync by its absolute path (/usr/local/bin/rsync).

So let's compare ... 3.0.5 with no symtimes to 3.0.6 with symtimes. And bugfixes of course. But still Protocol 30, and no significant changes since those directions.

Hmm. Well, I could mull this over with no way of knowing for sure, or I could re-run the unit tests in Backup Bouncer against this fresh compile!

sudo /usr/local/bin/rsync -aNHAX --fileflags --force-change --delete --stats /Volumes/Src/ /Volumes/Dst

sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst

Verifying: basic-permissions ... ok (Critical)

Verifying: timestamps ... ok (Critical)

Verifying: symlinks ... ok (Critical)

Verifying: symlink-ownership ... ok

Verifying: hardlinks ... ok (Important)

Verifying: resource-forks ...

Sub-test: on files ... ok (Critical)

Sub-test: on hardlinked files ... ok (Important)

Verifying: finder-flags ... ok (Critical)

Verifying: finder-locks ... ok

Verifying: creation-date ... ok

Verifying: bsd-flags ... ok

Verifying: extended-attrs ...

Sub-test: on files ... ok (Important)

Sub-test: on directories ... ok (Important)

Sub-test: on symlinks ... ok

Verifying: access-control-lists ...

Sub-test: on files ... ok (Important)

Sub-test: on dirs ... ok (Important)

Verifying: fifo ... ok

Verifying: devices ... ok

Verifying: combo-tests ...

Sub-test: xattrs + rsrc forks ... ok

Sub-test: lots of metadata ... ok

OK, that's pretty awesome! A complete pass! Now I wonder how much of that is because he knew what flags to use with rsync. So, out with the lart (the last MacPorts test in /opt/local/bin), in with the new.

sudo port -f uninstall rsync

sudo port -f install rsync

/opt/local/bin/rsync --version

rsync version 3.0.5 protocol version 30

Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.

Web site: http://rsync.samba.org/

Capabilities:

64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,

socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,

append, ACLs, xattrs, iconv, symtimes, file-flags

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you

are welcome to redistribute it under certain conditions. See the GNU

General Public Licence for details.

All the same capabilities as what I just compiled. Re-test!

sudo ./bbouncer clean /Volumes/Dst

sudo /opt/local/bin/rsync -aNHAX --fileflags --force-change --delete --stats /Volumes/Src/ /Volumes/Dst

sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst

Verifying: basic-permissions ... ok (Critical)

Verifying: timestamps ... ok (Critical)

Verifying: symlinks ... ok (Critical)

Verifying: symlink-ownership ... ok

Verifying: hardlinks ... ok (Important)

Verifying: resource-forks ...

Sub-test: on files ... ok (Critical)

Sub-test: on hardlinked files ... ok (Important)

Verifying: finder-flags ... ok (Critical)

Verifying: finder-locks ... ok

Verifying: creation-date ... ok

Verifying: bsd-flags ... ok

Verifying: extended-attrs ...

Sub-test: on files ... ok (Important)

Sub-test: on directories ... ok (Important)

Sub-test: on symlinks ... ok

Verifying: access-control-lists ...

Sub-test: on files ... ok (Important)

Sub-test: on dirs ... ok (Important)

Verifying: fifo ... ok

Verifying: devices ... ok

Verifying: combo-tests ...

Sub-test: xattrs + rsrc forks ... ok

Sub-test: lots of metadata ... ok

Well, whaddaya know ... it's not the new version so much as know what flags to send to rsync. So how does Apple fare with those flags?

sudo /usr/bin/rsync -aNHAX --fileflags --force-change --delete --stats /Volumes/Src/ /Volumes/Dst

rsync: -aNHAX: unknown option

rsync error: syntax or usage error (code 1) at /SourceCache/rsync/rsync-35.2/rsync/main.c(1333) [client=2.6.9]

/usr/bin/rsync --help 2>&1 | grep -- "-a\|-N\|-H\|-A\|-X"

-a, --archive archive mode; same as -rlptgoD (no -H)

--append append data onto shorter files

-H, --hard-links preserve hard links

--delete-after receiver deletes after transfer, not before

--address=ADDRESS bind address for outgoing socket to daemon

-E, --extended-attributes copy extended attributes

OK, so I'll whittle that down to what Apple's rsync will accept, swapping the X for an E (depending on version as to which flag means extended attributes).

sudo /usr/bin/rsync -aHE --delete --stats /Volumes/Src/ /Volumes/Dst

copyfile(70-extended-attrs/.._symlink-with-xattrs.4GgbUd,70-extended-attrs/symlink-with-xattrs, COPYFILE_UNPACK) failed:62

rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-35.2/rsync/main.c(992) [sender=2.6.9]

sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst

Verifying: basic-permissions ... ok (Critical)

Verifying: timestamps ... ok (Critical)

Verifying: symlinks ... ok (Critical)

Verifying: symlink-ownership ... ok

Verifying: hardlinks ... ok (Important)

Verifying: resource-forks ...

Sub-test: on files ... ok (Critical)

Sub-test: on hardlinked files ... FAIL (Important)

Verifying: finder-flags ... ok (Critical)

Verifying: finder-locks ... FAIL

Verifying: creation-date ... FAIL

Verifying: bsd-flags ... ok

Verifying: extended-attrs ...

Sub-test: on files ... ok (Important)

Sub-test: on directories ... ok (Important)

Sub-test: on symlinks ... FAIL

Verifying: access-control-lists ...

Sub-test: on files ... ok (Important)

Sub-test: on dirs ... ok (Important)

Verifying: fifo ... ok

Verifying: devices ... ok

Verifying: combo-tests ...

Sub-test: xattrs + rsrc forks ... ok

Sub-test: lots of metadata ... FAIL

However, since -a and -H are incompatible according to --help and hardlinks failed, let's try again.

sudo ./bbouncer clean /Volumes/Dst

sudo /usr/bin/rsync -HEr --delete --stats /Volumes/Src/ /Volumes/Dst

skipping non-regular file "10-symlinks/broken_symlink"

skipping non-regular file "10-symlinks/link2broken_symlink"

skipping non-regular file "10-symlinks/symlink1"

skipping non-regular file "10-symlinks/symlink2"

skipping non-regular file "10-symlinks/symlink3"

skipping non-regular file "15-symlink-ownership/symlink1"

skipping non-regular file "15-symlink-ownership/symlink2"

skipping non-regular file "15-symlink-ownership/symlink3"

skipping non-regular file "70-extended-attrs/symlink-with-xattrs"

skipping non-regular file "90-fifo/some-fifo"

skipping non-regular file "95-devices/devvn0"

skipping non-regular file "95-devices/devzero"

I think we know that will be ugly, but let's look.

sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst

Verifying: basic-permissions ... FAIL (Critical)

Verifying: timestamps ... FAIL (Critical)

Verifying: symlinks ... stat: ./symlink1: stat: No such file or directory

FAIL (Critical)

Verifying: symlink-ownership ... FAIL

Verifying: hardlinks ... ok (Important)

Verifying: resource-forks ...

Sub-test: on files ... ok (Critical)

Sub-test: on hardlinked files ... FAIL (Important)

Verifying: finder-flags ... FAIL (Critical)

Verifying: finder-locks ... FAIL

Verifying: creation-date ... FAIL

Verifying: bsd-flags ... ok

Verifying: extended-attrs ...

Sub-test: on files ... ok (Important)

Sub-test: on directories ... ok (Important)

Sub-test: on symlinks ... FAIL

Verifying: access-control-lists ...

Sub-test: on files ... ok (Important)

Sub-test: on dirs ... ok (Important)

Verifying: fifo ... FAIL

Verifying: devices ... FAIL

Verifying: combo-tests ...

Sub-test: xattrs + rsrc forks ... ok

Sub-test: lots of metadata ... FAIL

Pfft, hardlink still fail, but so do things that passed before!

If you're going to use rsync, I recommend using MacPorts or compiling the latest version yourself. Looks like Apple needs to add the fileflags and crtimes patches and update to a protocol version 30 rsync. But until then, MacPorts is easy!

DNS: dispatch shutting down due to TCP receive error

I finally got annoyed at the "dispatch 0xhexcode: shutting down due to TCP receive errors: connection reset" in the logs on one server, so I asked Google.

The best answer seems to be from SANS ISC, but even the simple "the other end shut down prematurely" is enough to point out it's probably not a problem and it's not on the server's end.

Solved!

Aha! I solved a problem I've been wrestling for a while, to merge XML files!

1. Use cli, not this nifty GUI; follow the example on the source web page, but modified for the java from TestXSLT.
2. And use CDATA so I don't have to urlencode carriage returns and possibly other nonsense (after noting that urlencoding helped the cli run through its paces).

Still not as user-friendly as what I envision, so I'm not going to be too specific yet ... but since I can pack several of my tricks for work into XML, if you can pick and choose what to merge together, then who needs me to do it for you?

On Duplicate Key Update

This morning I was trying to copy a permissions-customized mysql.users table from the production server to its backup and development servers. Of course in about 2 seconds, I get a message of "Duplicate entry whatever for key numever" because these servers are already pretty similar. So with a little Google, I found the INSERT ... ON DUPLICATE KEY UPDATE syntax that should help me with the mysqldump I want to restore. Hopefully I can get myself out of the woods.

How To Compile

I'm sure everyone with Unix experience has their own cherished best practice to compile software from source. I'll try to tone down the soapbox.

What set me off was a TidBITS article on DNS (at work, we compiled that patch as soon as it came out). I was ready to yell at my computer screen! They got it right about the need to patch right away, but wrong on how to do it well.

The very basic process to compile from source is:

./configure

make

make install # as root

Well, I like to log the results of those three commands; if it fails to build, I've got something to study later without worrying about my scroll buffer. I also like the visual feedback of how the compile is going, so I also want to see the output as well as logging it to a file. The reason why make install is its own step is so that you can follow the principle of least privilege. My method does all of those things.

./configure | tee configure.out

make | tee make.out

sudo make install | tee make-install.out

Feel free to choose your own file names, or switch to root instead of using sudo (potentially more secure, but longer). However you do it, just keep in mind that logging and least privilege are Good Things.

Busy with DNS

So we all heard about the DNS flaw announced by DoxPara. Boy, was that a busy week for us! Turns out, the secure version of BIND has this little problem with CPU load, accompanied with complaints about file descriptors, above a certain number of DNS queries per second. Ouch.

The immediate key to get it under control was to add ulimit -n 4096 to named.conf so that BIND would use more of the available file descriptors. The fix with more breathing room was to install the next beta version of BIND that has better performance. We've been out of the woods since then, and we're no longer expecting another shoe to flatten us.

In fact, now we can relax while those who didn't patch have discovered that the flaw has been discovered before its scheduled public announcement ... yikes!

Redirecting stderr

I have some scripts that echo warning messages that really should go to stderr instead of stdout. The basic form is

echo "error" >&2

but I've twice gotten frustrated when I've tried to get fancier than that (not with echo, but with other commands). The essential problem is that the redirecting to stderr needs to be the last redirection. So my quick example is

echo "error" > /dev/null >&2

where redirecting to /dev/null or a file must come before the stderr part.

Hope this saves some time!

tabs and sed

I need to remember how to do this trick, using sed on literal tabs, which you can usually get by typing ^v. Control-V, Tab allows sed to operate on tab characters. Yes.

Passing Environment Variables to sudo

Aha! I can pass environment variables to sudo!

sudo env ARCHFLAGS='-arch ppc' python setup.py install

Or, more to the point,

sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH sybase

iCal filtering

I wanted to pull out just the 2008 entries from a calendar in iCal (ics) format, and import those into a calendar of my own.

At first, I went looking for an IcsToCsv converter (click on Attachments to see the Python code; there's Ruby code for ics2csv too). My plan was to use grep to match the 2008 lines after the CSV format put the entire entry on one line. Next I realized that it was much easier to use Smultron for the CSV conversion since all the event entries were uniform.

Then I noticed I could go straight to the end game, without CSV, with judicious use of grep to include context before and after the match ...

grep -B 1 -A 4 "DTSTART:2008" Exported.ics

So the Sample file I wanted to import could be created this way:

sed -n '/^BEGIN:VCALENDAR/,/^BEGIN:VEVENT/p' Exported.ics | sed '$d' > Sample.ics

grep -B 1 -A 4 "^DTSTART:2008" Exported.ics | grep -v -- "--" >> Sample.ics

tail -5 Exported.ics | sed -n '/^END:VEVENT/,/^END:VCALENDAR/p' | sed '1d' >> Sample.ics

The middle command is the good one; the first and last commands are safer to do visually because they assume that all of the iCal events are formatted just like the ones I scanned ... and it can be dangerous to assume that much!

If you have problems importing, check the line endings. On my Mac, I ended up with ^M (as seen with the vis command, or with cat -etv). You can remove those with this command:

cat Sample.ics | tr -s '\r' '\n' > Clean.ics

(If you want to preserve blank lines, don't use -s.) My import went fine. I love my command line text manipulation tools!