What I learned rewriting old Perl scripts in Go

As part of my project to learn the Go programming language, I decided to rewrite some of my old Perl scripts in the newer language to see what was better, if anything was worse or if things stayed the same.

The scripts are not very glamorous, but are useful for my set up. One is for generating scripts that wrap around rsync in order to keep two directories in synch. The other (which is closely related) is for keeping track of which files have been marked for deletion and keeps deleting them if they reappear. Once one starts to copy files back and forth between directories, deleting files becomes a little bit complicated as the file you delete will keep reappearing unless you delete it everywhere. The script I wrote them before Dropbox and the like had come on the scene. I keep them around as I don’t like to have the only copy of my photos and videos in a folder managed by someone else. They are also useful for folders that have too much content to be stored practically in Dropbox, like virtual machine files, and synching folders on servers via SSH.

I have been meaning to update the scripts and change their behaviour for a while, so a rewrite is a good time to ditch some obsolete features. The synch script generator used to be able to generate batch files to run on Windows. Now, it just creates bash scripts. This is fine (for me) as, on Windows, I use Cygwin for rsync, so bash is not a problem for me as a dependency.

Another change is that the script generator used to silently skip generating a script if a file was in its destination. The thinking was that the generated scripts could be considered scaffolding for more capability that could be added by adding to or amending the scripts. The script should not overwrite these changes. Experience, here and elsewhere, has taught me that this is a nightmare to work with. Generated files should be regeneratable at any time. In light of this, the generated scripts are marked with a warning comment at the top saying that they have been autogenerated and are blasted out of the way when the program is run again, potentially as part of a scheduled task.

The script generation program is at:

https://github.com/robert-impey/generate-synch-scripts

The second program is for making sure that files stay deleted when two folders are synched as described above. This had been at

https://github.com/robert-impey/stay-deleted

However, as the project grew and I need to move code to more than one file I learned that package names in Go should not contain punctuation, so I started a fresh project and moved my code there:

https://github.com/robert-impey/staydeleted

My new version of the program is working correctly. I’ve also added a few new features like deleting old metadata files and an option to run repeatedly at random times, which is useful for running as a scheduled task.

The most immediate advantage that I have found for this script is the ease of distributing a binary generated by the Go compiler. The nature of the software requires that the program runs on every computer in the system, for my set up that includes Windows, macOS and Linux. Getting a perl script with a few library dependencies to run on all those computers was a pain. With Go, it’s been trivial so far.

In order to have a more modern command-line interface, my program now uses spf13’s Cobra library:

https://github.com/spf13/cobra

This has forced me to reorganize my code to some extent. This has been a good thing. However, I’m still very much at the beginning of my Go learning journey. Whilst the program works; the code is a bit of a mess. I’m not sure how to apply the clean code principles (such as SOLID) that I use in my day job writing C# to Go. Perhaps this is one of the main advantages of aiming to be a polyglot developer. Everyone should aim to write readable code in every language. We discuss and come to an agreement about design principles to do with how to use the language’s features in order to make code more readable. When one moves to a new language, you might not know how the language goes about implementing those features or those features don’t exist. What happens then to those design principles? Can they be reformulated for the new language?

Make Turning Windows Off and On Again Great Again

There’s a joke about all Windows related problems being solvable by turning the computer off and on again. It’s one of the things that works most reliably about the OS. Well, even this has been broken by Microsoft in the Windows 10 Fall Creators Update.

When you get to the point where you have more than a dozen virtual desktops running with many different programs, your computer (not to mention your brain) will start to get overloaded. At times like these, I just want to hit refresh (to coin a phrase…) and get back to an unmussed system. However, a new “feature” in the Fall Creators Update of Windows 10 will try to reopen as many of the programs that you had running beforehand as possible. Not all programs will get restarted. Tiny programs like Notepad++ stay unloaded, but bloated behemoths that hang for 30 seconds to a minute to load (like Excel and Visual Studio) get restarted, maybe with the files that you had opened, maybe not. If you were using virtual desktops beforehand and had carefully separated your programs by task, all the programs will be in the first one but you will still have several empty and useless virtual desktops open. Instead of getting a fresh start on a clear desktop, you are left waiting for programs to load and needed to close all the virtual desktops. Unfortunately, there’s no setting to just turn off this rather half-baked behaviour.

However, a little PowerShell can reduce the pain and shutdown the computer more thoroughly.

I don’t want to accidentally trigger this, so my script starts with a confirmation:

Write-Output "This will close all your virtual desktops and shutdown the computer."
$Confirm = Read-Host "Hit Y to confirm."
if ($Confirm.ToLower() -ne 'y')
{
    Write-Output "Doing nothing"
    exit
}

Write-Output "Shutting down..."

Closing all the other virtual desktops can be achieved with by wrapping around MScholtes’ VirtualDesktop:

while ($true)
{
    $CountCommand = "$($VirtualDesktopExe) /C"
    $CountOutput = iex $CountCommand
    if ($CountOutput -match 'Count of desktops: (\d+)')
    {
        $Count = $Matches[1]

        if ($Count -gt 1)
        {
            Write-Output "You still have $($Count) virtual desktops."
            iex "$($VirtualDesktopExe) /R"
        }
        else
        {
            Write-Output "Only one virtual desktop at this point"
            break
        }
    }
    else
    {
        Write-Error "Unable to parse result of command '$($CountCommand)': '$($CountOutput)'"
        exit
    }
}

This simply invokes the program with the “/C” flag (count) then calls the program with “/R” (remove) until only one virtual desktop remains.

After that, the script just invokes the old fashioned shutdown command from Ramesh Srinivasan’s article above about this new feature.

shutdown.exe /s /t 0

On my desktop, I have a link to a script that invokes the PowerShell script with the location of the VirtualDesktop.exe file on my machine:

PowerShell -command "F:\Robert\Dropbox\local-scripts\_Common\CloseVirtualDesktopsAndShutdown.ps1 -VirtualDesktopExe F:\Robert\Dropbox\executables\Windows\x64\VirtualDesktop.exe"

Now I know that I will get a fresh start when I turn on my computer again. I’m not really sure why this isn’t the default behaviour.

One liner to show logs without Rotated Backup files

I’ve been looking at the Apache log files on a web server this morning. There are many virtual hosts on the machine and the log rotation scripts have created many numbered backup files of the logs. To make the directory listing more readable, I have been using the following one-liner:

ls -l /var/log/apache2 | grep -Ev -e '[[:digit:]]+(\.gz)?$'

This will only display the current log files, assuming that /var/log/apache2 is the directory in which you store your Apache logs and that you do not store any other files there.

I hope it helps.

New Domain for the Blog

This blog will now be hosted at

http://www.reversing-entropy.com/

to reflect the title. Any requests to the old address:

http://impey.info/blog/

will be redirected to the new address.

Percidae

For the last few weeks I have been looking for a job as a programmer. One of the jobs that interests me involves writing VB.net. Many of my friends who are also programmers have howled with anguish at the mere mention of this technology.

In order to get a better idea of how the language works, I’ve written a small program for editing the PATH environment variable on Windows machines:

http://code.google.com/p/percidae/

I find myself editing this variable on various Windows machines pretty regularly. Every time, I also am annoyed by the small text box in the dialog in the Control Panel.

So far I’ve enjoyed the experience of writing with Visual Basic and can’t see what all the fuss is about. Having written PHP for years, I’m used to ignoring the comments of language purists. I’m much more interested in getting something working than any imagined superiority of different languages.

One-liner to SVN delete files that you have already deleted.

The following one line shell command allows one to delete files from your file system and then removed them from Subversion.

svn stat | grep '^!' | sed 's/^! *//g' | xargs svn delete

One-liner to make sure svn updates work

I’ve been having problems getting Subversion updates to complete reliable of late. I have an enormous working directory called ‘programming-projects’ that is basically a large list in svn:externals. It’s useful to be able to go to the root of that working directory and update everything at once. This is especially useful for checking the development versions of projects that are using the latest versions of Haddock CMS.

However, normally before all the directories have been updated, one of the external servers will return a 502 error (or similar) and the process will die. The best solution that I’ve found so far is to run the following command:

perl -e '$r = 1; while($r) { $r = system("svn up") } '

It simply keeps calling svn up until it is successful. It’s not very elegant, but it works. Is there an argument that you can give to Subversion that will achieve something similar?

CLI Script to allow access to a directory in Haddock CMS projects

I’ve added a little script that allows developers to allow access to a directory that is relative to the project root directory of a Haddock CMS project:

PublicHTML_AllowAccessToDirectoryOnTheServerCLIScript

I refactored a bit of existing code from:

PublicHTML_RestrictAccessToDirectoryOnTheServerCLIScript

into an abstract CLI script class:

FileSystem_ExistingDirectoryRelativeToProjectRootCLIScript

If you ever need to write a script that does something with a directory that exists and is relative to the project root, then extending this class would probably be a good place to start.

How did Microsoft get Vista so wrong?

Before anyone accuses me of being a Linux bigot, I would like to say that I’ve been frustrated by Ubuntu lots of times. Wireless networks on laptops have always been a bit of a bugger and my latest install on a partition of my laptop has been no exception. Getting an EEE PC has shown me how good Linux on a laptop can be, if it’s set up right by the manufacturers. Ubuntu does quite a good job at this but it’s certainly not for the impatient. Dell, Samsung, anyone, please start selling more Ubuntu laptops with all that boring driver nonsense sorted out!

Working with Debian servers at the command-line has never been anything but an unalloyed pleasure. I have a extremely complicated set of tasks that I want to achieve and the stable version of Debian has always done them quickly and painlessly. Some stuff takes research. I’ve no idea how much of my career has been taken up with reading tutorials on the syntax of UNIX config files, probably more time than I’m going to get back. But once you know something and it works it works well. On servers (which, at a glance, are indistinguishable from their counterparts from the 1970s), the bottlenecks have always been my intellect, knowledge and imagination.

And then there’s Vista.

At first I thought that it was a brilliant. Good look, nice fonts, WinKey+Tab 3D funkiness and so on. But then you use it and before long you need a shot of whiskey just to calm your nerves.

If I access an FTP server (even on a cheap shared host) or SSH daemon, logging on and moving from directory to directory is quick. Most programs, including Nautilus out of the box on Ubuntu, allow you to store previous connections. XP used to remember the SMB shares that I had accessed. However, in Vista, every time I go to the network window in the start menu, the list has to be refreshed. Why? And does this takes so long? Does the computer ping the whole of 192.168.*.* or something?

Eventually, you get a list of computers on the LAN. You start to move about but just going from one folder to another can take up to a minute. Eventually you get to a folder that just locks up the computer for a few minutes, Explorer tells you that access is denied and restarts Explorer.

You get a link to

http://support.microsoft.com/?kbid=937097

which tells you that an error occurred and gives you information on how to load up the event viewer that also tells you that an error occurred. Great! I guess that I better contact my system administrator.

I had hoped that the Vista service pack would sort this sort of nonsense out but it hasn’t.

I’m loath to spend an evening hacking away at config files on the Ubuntu partition of my laptop just to get the sodding wifi adapter to work but anything’s gotta be better that the soul destruction that is using Vista all day everyday.

People talk about Cognitive Surplus:

http://jeremy.zawodny.com/blog/archives/010218.html

I guess that any system where the bottleneck isn’t your intellect, like Vista and Ubuntu some of the time, then the thoughts that should be going into your work end up getting clogged. Hence, the need for hard liquor…

YAWAF?

I run this blog using Word Press on a shared server. I could use one of a number of free blogging services like blogger.com and save myself the effort of having to update the software every now and again and deal with the other admin tasks that running a blog this way. I don’t because I just the tiniest bit paranoid. I don’t want to sign everything other to Google; although I do already use Gmail, Google Apps, Picasa, Google Code and Google Analytics so it’s not like a could be much more dependent on them. For a second after looking at the list that I just wrote I got really paranoid. And then I felt resigned – maybe the blog should go on Google as well.

I just took a look at:

http://code.google.com/appengine/

Part of me worries about this and part of me is resigned to it. Even a little seduced by it. The scalability, free hosting and (I assume) increased reliability are very attractive.

I guess that it won’t take over the enterprise web app market for a while but I guess that that has to be a target. But how much does one want to lock oneself into a company for this sort of thing? Can we use deploy apps on other servers with other companies?