Author: Fergal Hainey

Followup roundup

Written by Fergal Hainey

I am thinking of reviving this blog. I am hearing from all angles why I should be publishing: in case what I share is useful, regular writing is supposed to be good for me, and because maybe “personal branding” is a fairly good idea after all.

The first thing I want to do is a draw a line under the old topics from 2012 while I was still at University of Aberdeen. I actually enjoyed reading them again, the topics are so unrelated to what I do now that they aren’t cringe worthy.

Boombox

My JVC ’80s boombox has had a good run. It has developed a hum on the tape deck that I can’t fix with my usual routine of poking the contacts with a spoon to ground them and twiddling the knobs to get static and dirt out. I don’t want to have to throw it away next time I move flat, so writing this has spurred me to contact a repair shop before I finished this paragraph.

In the end I used the FM transmitter, tape deck connector, and blank tapes pretty evenly depending on the circumstances. The FM transmitter if I needed to use my phone at the same time, the tape connector for better sound quality, and tapes for relaxing.

I have noticed an increased number of battery-powered small Bluetooth speakers that actually sound good. Finding something portable that sounded good without being too expensive is what drove me to buying a vintage boombox in the first place. The UE BOOM is the one I’ve heard the most of and I’m impressed by it. Some others sounded OK and there have been others that I didn’t think sounded good enough, but I didn’t take note of what they were. They seem to be in the £100 plus range still.

Node.js filtering proxy

I was really happy with how fast it is to get up and running with Node.js. I realise that forwarding all of your browser’s traffic isn’t a high load situation, but is more than Flask’s debug server would take with default settings for example. I appreciate that Node.js’ standard pattern worked so well out of the box, it’s evented and has a fast VM. Python is my first language now and I love it for how it encourages team maintainability, but often needs extra consideration to reach production performance. Python’s async syntax and PyPy should bridge the evented and VM gaps respectively. The new language features in ES6 should make JavaScript more attractive. Exciting times! I like to think of ease of maintenance by a team and performance as different facets of working at scale.

The code I wrote probably doesn’t work with 4OD anymore. I don’t have a television now so I don’t see adverts for shows, so I haven’t used any of the catch up services in years. It was a fun exercise though and gave me something to talk about in a job interview.

SD to HDD copier

I never tried it. I don’t hear the complaint of full SD cards so much anymore. I think the cloud and even lower SD card costs have solved the problem. I’m still excited about what new products these education focused electronics will lead to. The Arduino is still a great microcontroller. The Raspberry Pi is a great embeddable full computer and the new models are really powerful and still cheap. The new BBC micro:bit is awesome for experimenting with the components in modern mobile devices. I was worried that the micro:bit would be stepping on the Raspberry Pi’s toes, but after getting one at EuroPython 2016 I was won over. The micro:bit definitely has a focus on mobile devices, which makes a lot of sense as most people’s first computer is a mobile.

Universal repair USB stick

I feel guilty that I didn’t document my steps properly. I still have the one USB stick, but I’ve not had to use it in years. Modern Macs and Windows computers have built-in repair partitions. Upgrading to the latest OS is a lot more common now the upgrades are free and can be downloaded, so there seems to be fewer people with messed up systems. I myself hardly ever use a laptop anymore and do most of my work on a tablet. When a tablet goes bad, you just wipe it. That sounds like the worst case and evokes memories of reinstalling Windows. The reality is all the apps, settings, and documents are backed up on the cloud and starting from scratch again isn’t a big deal. With app stores for most platforms and Microsoft and Apple both peddling cloud storage, desktop could be going the same way. I’ve been considering trying to Dockerise my whole dev environment so even trickier installs are easier to reproduce. I’ve not seen anyone using Docker that way yet so could be interesting.

Clipboard sharing with Dropbox

I got a lot of use out of my little Dropbox app. Eventually messaging apps got better at accepting different sorts of pasted content and I stopped using it. I read the help page linked from my Public folder and it appears that only Pro and Business users can use Public folder simple URLs now, so my app may no longer work. If I were to make it now I’d probably have to use Dropbox’ API to make tokenised URLs. The help page also says that as of 2016-10-03 HTML documents won’t be rendered in a browser, I guess they’ll change the content type. My app relied on that for sharing rich text content.

The future

I do hope to keep this blog somewhat up to date now. Everyone in the tech industry has moments when they feel they were the first one to get a particular combination of parts working together, so I will try to share those times at least. I work mostly with Python, Docker, and AWS now so things could be more focused in that direction. That said I want to start making and computing as a hobby again not just work, I think about it so much.

Share clipboard with Dropbox

Written by Fergal Hainey

Dropbox is pretty good. Like anything, I wouldn’t trust it with everything important but it has features that make it more useful than just a synced folder.

One feature I like is being able to share the URL of something in your dropbox public folder easily. On a Mac you can right click a file in the folder and select “Copy Public Link”. You can then paste that link into a chat or e-mail or whatever.

It’s so useful I use it for stuff that isn’t even on dropbox already. I’ll copy something, paste it to a file, then move the file to dropbox and get the link. Screenshots is a particularly often used example. It’s so close to perfect for sharing things, it makes me want it to be just a little better to complete it. If there was a way to turn what’s in my clipboard into a dropbox URL without leaving the application I’m in, that would be perfect. I’m sure I could bridge that gap.

One of the coolest things about Mac OS that makes me surprised it hasn’t been adopted more by computer science types, is that pretty much every graphical application (even 3rd party) and system service can be scripted. It’s been going on since even before OS X and is called AppleScript. With little developer effort, objects and methods can be mapped to the scripting language. AppleScript is perfectly capable of accessing the clipboard, turning it into a file with the appropriate type, then putting that file somewhere. Dropbox public URLs are very simple, and are based on a user ID stored in a simple config file and the file name, so are easy to synthesise.

I made an AppleScript to do all that when run, and then packaged it as an Application. I can now use Spotlight (OS X’s system search, accessed with a key combination from any app), which prioritises applications to make them easy to launch, so I can now do this task within seconds without even needing the mouse.

I’ve made the application (which is its own source) available on GitHub at https://github.com/Riprock/Paste-Dropbox.

Icon copy

Proxy followup

Written by Fergal Hainey

When doing my work on the filtering proxy, I did think about whether it could be made to work with HTTPS as well. The only useful plain text part of a HTTPS connection is the domain, so that seems to be a limiting factor on what I could do, just block domains. Obviously the rest of the data going through the proxy would be encrypted, but as my computer was the sender and intended recipient, surely I could decrypt it. As it turns out, Chrome (not my go to browser) can be configured to output a file containing they keys it uses in HTTPS. I discovered while using wireshark on another project. Wireshark can use this file to let you inspect HTTPS streams. Maybe my proxy could be updated to use this file (and parallels in other browsers if they’re there) to allow filtering for HTTPS.

Bootable USB stick followup

Written by Fergal Hainey

I discovered that using a GPT partition scheme meant that my stick wouldn’t work with older systems, which is a shame because I had a partition for using like a regular USB stick which is useful for transferring files around.

As it happens you can have disks that are both GPT and MBR, as detailed at http://www.rodsbooks.com/gdisk/hybrid.html (note that that guide is written by Rod Smith, the maintainer of rEFInd mentioned is my last post on this, and also gdisk, which I used for making hybrid MBRs before I realised it was also one of the things iPartition could do).

Another thought I had was: what about these news PCs that will be using UEFI (an updated and more standardised EFI)? I have some theories about being able to put the right file (say /efi/boot/bootx64.efi, which is a standard fallback location) in a partition by itself, with bootx64.efi being a GRUB EFI application made to use my existing HFS+ partition for it’s modules. Unfortunately I haven’t got any computers to test this on.

A bootable repair USB stick for Macs and PCs

Written by Fergal Hainey

Sounds useful. Should be a lot easier to do than it is. This has been an ongoing project of incremental improvements and sudden inspirations, I don’t think I’ll be able to document it all.

OK start from basics. To be bootable you need something to boot. GRUB is a bootloader that will work on PC and Mac. OK good, how to make a computer boot it. Using the age old method of BIOS booting. Put a little bit of machine code on the first block (512 bytes) of the disk, BIOS computers will run this, and it should do things like find any more machine code it needs and list and boot operating systems on the computer. GRUB needs to be installed to work on BIOS computers in sort of 3 parts. That first bit of boot code, which finds more boot code on the disk at a static offset which needs to be preprogrammed in the first part. Because there will be some sort of partitioning scheme like MBR that takes up the rest of the disk, the scheme will have to leave some space somewhere for the 2nd part of the machine code. Once GRUB has got all that together it should have enough functionality to access a volume/partition on the disk and load the rest of its modules and settings from the filesystem.

So I made an MBR partition scheme with enough enough free space on my new USB stick and set about installing grub on it. The grub distribution running on my Debian virtual machine was designed to tune the grub installation on the computer it was running on. I extracted elements from it to create a more generic set of grub installation scripts. And reading the manuals and doing some tests I managed to get an image of my USB stick to boot in a VM. Unfortunately I don’t have a spare computer lying around the test the actual USB stick.

GRUB has a nice bit of functionality which can make CD ISO9660 images (.iso) into loopbacked filesystems, and it is good at booting linux with just a kernel file and an initial RAM disk file. So any linux distribution which can be passed boot parameters to make itself aware of this and do a similar loopback mount of the iso (like SystemRescueCD and Debian install discs) can be booted like this without having to make separate partitions for them, which is nice.

A Windows install would be nice for fixing (read reinstalling) Windows stuff too. The one I have a license for is Windows 7 Professional 64 bit. I don’t think I’ll be able to do this with just an iso image. GRUB is not so good at booting windows, it relies on chainloading: being forwarded an address with more machine code to run to boot the operating system. I copied all the files on the windows install CD to a partition on the memory stick so most of the stuff it needs should be there. As far as I can tell there’s nothing on my Windows 7 install CD image to make the boot code for arbitrary partitions on a USB stick. Luckily I discovered ms-sys, which has recently added being able to to write what it calls an NTFS boot record, which I hoped I would be able to bootstrap the files on. Like the boot code at the start of a disk before the MBR partition table this boot record sits at the start of a partition, before the NTFS filesystem structure. Chainloading this boot record worked to my great surprise, I hadn’t found anyone else on the internet using this method.

So I had a USB stick that would theoretically work on a BIOS PC. Good start. Now lets’ looking into Macs. Macs don’t use BIOS. Bad start. They work with a sort custom version of an outdated EFI (Extensible Firmware Interface, designed to replace BIOS) spec. Firstly it required a GPT partition scheme to the disk. OK I can do that. Then there are two ways to specify what to boot, either by setting flags and details in the computer’s NVRAM, which require a system running which can do that (like the “bless” command on OS X) so is not ideal for making a repair disk, or with headers in Apple’s HFS+ filesystem. There are no other ways. So GRUB will have to use EFI, on a GPT partitioned disk with a HFS+ filesystem, very different to the BIOS, MBR, FAT32 setup I have.

Let’s see about reconciling the two. BIOS systems can work with GPT, there’s still the 1 block gap at the start of the disk. The GPT filesystem can then set aside some free space for the rest of GRUB. I can keep my old FAT32 partition with grub BIOS stuff in it. As long as there is a HFS+ partition with the right headers in it pointing to an EFI application, a Mac will be able to boot it. Grub can come in an EFI version, but it doesn’t do BIOS so I’ll need both versions on the stick. The Mac version will need its own modules, but should share the config file containing a list of operating systems and how to boot them with the BIOS version, GRUB config files have the “configfile” directive for this and grub can traverse different partitions and filesystem formats easily, as long as you include the right modules.

Around this time I learned that the Mac app I’ve had for a while, iPartition, was much more capable than I thought originally. It excels in nondestructive moving and resizing of partitions and changing the partition scheme and can give complete control over where on the disk partitions go. It could even easily create the BIOS boot partition GRUB needed on GPT disks. It gave me functionality I was expecting to need a collection of obscure command line tools for. I recommend it for anyone who needs to do more than basic disk partitioning.

At this point I had a USB stick that would boot in my bios based virtual machine, and would boot on my actual Mac. I consider that a success. I still can’t get Windows 7 install to work with EFI. You can’t chainload BIOS boot code from EFI. Another bootloader, rEFInd claims to be able to get into the Mac’s (and some other manufacturers’) CSM (Compatibility Support Module), but that may be cheating, and using two bootloaders would make things more complex. Windows 7 does support booting from EFI, and I’ve managed to make a little progress with the EFI app “bootmgfw.efi” on the install disk, but either it’s just doesn’t have the routines for the situation it’s in, or maybe I can change the BCD to make it work.

Boombox follow up

Written by Fergal Hainey

Just saw this on the Maplin website http://www.maplin.co.uk/low-noise-stereo-preamp-kit-220053 . A lot cheaper than a CMoy and has the right impedances. I could split my 1/8″ jack to DIN cable and put this in the middle with a 9V battery and it could work maybe.

My friend with the tapes also gave me an old stereo audio transformer from RS he said could work. Sounds like some sort of magic bullet, a passive one component solution to clean boosting. I may give it a go but am a bit worried about the arcane stuff about impedance matching and also that I may blow something in the boombox if I’m mistaken about it’s power output.

SD to HDD copier for video recorders

Written by Fergal Hainey

When on long kayaking holidays with friends with video recorders like GoPros, we either have to stop videoing because the SD cards fill up, or bring a laptop to empty them onto. Neither is ideal, especially as the laptop can run out of battery fairly early on.

It seems there used to be a market for portable SD card backer uppers, but they were priced for professional photographers, as they were the only ones who’d take so many photos that it would be more economical than buying another SD card. It’s hard to fill up a couple of modern SDHC cards with photos, so these backer uppers seem to have faded out. Now there is a new problem that video files take up a lot more space than photos, and for best results video recorders need to be left running so you don’t miss any of the action.

My first thought was that maybe this is something you could achieve with a microcontroller, like an Arduino. Low memory and data throughput makes this not very feasible. Although the task is simple, unfortunately it looks like this one of those things you do need more power for. This spurred me to think of the Raspberry Pi, the new development board that has been on the news as it intends to be an education platform. The specs make it look perfectly capable, and it should have a similar price and footprint to an Arduino. Sounds perfect!

Although I can’t get hold of one yet, I got to thinking about how it would work. It already has USB connectors, and can run a sort of Linux. Making scripts to copy things when there’s an SD card and a USB HDD plugged in will be easy with a Linux OS already on there. Thinking about the USB, I realised it may not be able to handle the current required of a HDD, even a portable 2.5″ one. From the specs it looks like running a disk from the USB ports would not provide enough power. Well, a lot of top end USB HDDs take mains power, so it’s possible to have a different power source. How about a USB hub which provides power, and plugs into the Raspberry Pi to provide data. A quick read over USB specs shows supplying a separate 5V power line to any device should be fine. And the Raspberry Pi and can take its power from USB, so the hub could power the Pi too, perfect. The addition of a voltage regulator could make the hub able to be powered by batteries, perfect for travelling.

So a Raspberry Pi, a USB hub, and a battery compartment. Should take up a lot less space in your bag, won’t run out of batteries, should be a lot cheaper than the old photo backer uppers, and you can supply your own HDDs. Considering the Pi has HDMI out, it could be plugged in to review videos too! The Pi runs its OS off an SD card, so it should be easily upgradable with new features. It does sound like a very feasible solution.

All this makes me wonder, could there be in the future a whole range of cheaply available gadgets, and you simply plug in your Pi and save the cost of all its components with each one. Simply add to the SD card  the data you need to run each gadget you have, do some hardware detection, and easily do updates and feature additions! This would be a great boon too consumers, not just people interested in tinkering with things.

A HTTP proxy with filters

Written by Fergal Hainey

This idea was sparked by Channel 4’s online player 4OD. It makes you watch adverts like you were watching a television, some before, some after, and even some in the middle. It makes sense I guess, they still have to get paid. Except all this is happening on my computer, so I’m sure I can do something about it.

First thought, ad blockers. Don’t work, it’s all stuff within a flash object I essentially still want to use (to watch the actual content) and all the ad blockers I can find are actually still pretty simple, they just block URLs.

Some WebKit “Web Inspector”ing and a bit of wiresharking later and I find the URLs of the adverts being shown. They’re all from CDNs and have complicated session URLs, so I make a general pattern for their URLs to give to an ad blocker. When the video files are blocked, the 4OD player declares there is a problem (which I guess there is really), and stops playing. So it’s going to be more complicated than that.

More wiresharking. Let’s find out where the flash object is getting these video URLs from. From an XML document, with a nice clear structure showing where the adverts are and when to play them. If only there was a way of just removing that <adverts> tag. A browser extension (like greasemonkey or Safari extensions) could change the content of stuff fetched by the browser, but the XML document is being requested and processed by the flash plugin, so that’s out. Some kernel extension to intercept network calls? Sounds messy and beyond me. A proxy, except a misbehaving one that alters content before it forwards it (like those stories of those American ISPs that do the opposite of what I’m trying to do, inserting ads)? That could work. Is there something that already does this? Maybe, but uncertain if it’s configurable enough, and it costs. So that’s out. Can I make one? Good question.

I’d been hearing a lot about Node.js, a platform based around javascript and designed to make the evented model of javascript (like AJAX) available for use with general programming, especially IO. It was a language I already had a lot of experience with, and had libraries for HTTP requests and responses. And the evented model was supposed to make stuff like this fast and efficient. So should be perfect.

The way proxies work is pretty intuitive, and there’s plenty of documentation about it. Making a simple one in node took less than 50 lines, and it worked! Adding filtering rules was simple, I didn’t need to configure my proxy, I could just rewrite it however I wanted! The first thing I tried was removing the <adverts> tag in URLs that matched the 4OD XML info document and it worked. The adverts simply disappeared! A bit of smartening up and I had an easy to use (for me) system to handle any task like this when I need it!

iPod to ’80s boombox

Written by Fergal Hainey

I first started looking into vintage boomboxes when a friend got one those BOSE iPod docks. I thought a small set set of speakers that sounded good and could fill a room was a great idea. My first thought was that the new wave of iPod docks were made small so they would be portable, except they had no batteries and no handle. All the ones that were actually portable were passive (driven by the iPod, so quiet and bad sounding) or very expensive and in sort of a wheeled flight case format.

I was getting into golden age hip-hop at the time so quickly thought of old boom boxes, but assumed they were all the massive ones you see on album covers and not up to todays audio standards. A bit of research proved there was actually a very comprehensive range of sizes in manufacture before they fell out of fashion. There is also a large boombox collectors community on the internet who have a lot to say about the sound quality. It turns out that a competitive market and a consumer demand for loudness resulted in boomboxes that I’m sure would outperform atleast passive speakers. I also realised that consumer audio equipment doesn’t include all the detail and new technology of studio equipment, and now that headphones and speakers are more of a commodity that gap has probably widened. I also have my own suspicions that a generation used to listening to poorly encoded MP3s on YouTube are probably less discerning of audio quality, and these iPod dock manufacturers have could have taken advantage of that to cut costs.

So I took to eBay and am now the owner of a JVC RC-656LB for around £40! It’s about a foot and a half in length and has a very classic ’80s look to it, which makes sense as apparently they started manufacture of that model in 1982. I first tried it by tuning the radio and how loud and clear it could get really surprised me. It was a much better system than I was anticipating, especially the bass response!

002vf

To begin with I was content to make tapes. I inherited a Hi-Fi with a cassette recorder from my old flat and had some fun learning about the different types of tapes and recording methods, such as Dolby noise reduction. I bought the last box of cassettes from my local Currys and burnt through them within a month or so. I quickly found the attraction in making mix tapes. Selecting tracks and matching the lengths of each side was very cathartic, I soon had more mix playlists than cassettes. I put some whole albums on  tape too, and was taken back to the days of CD players, where you couldn’t skip around between albums and artists. Generally limiting what I could easily play gave me a much better appreciation for the tracks I was choosing, and also made me think a lot more about what tracks to put on tape, what music I really liked and could listen to all day.

I was lucky enough to have a friend who had a lot of PA equipment going back decades, including boxes and boxes of TDK chrome tapes. Chrome tapes (Type II) have a different frequency response to normal ferric (Type I) tapes,  so the recording and playback equalisations could be changed to give a better dynamic range in the high frequencies (assuming your tape player can adjust for them). Due to his generosity I now have a practically unlimited supply of tapes of a higher quality then the ones I can find in shops!

The boombox was going everywhere, the kitchen, shower, garden, picnics, beaches and camping trips. It could last a good few weeks of every day use on 8 D cell batteries (around £10 as long you can find a bulk pack of 8, unfortunately it chewed through cheap batteries when I tried so they weren’t an option). On longer trips my friends contributed to battery costs when it ran out, so it wasn’t too uneconomical. While at home it runs of the mains. Being the only one who could make tapes I was enjoying having complete control of the music. One thing I hate is skipping between different tracks to appease different people, especially before a track has even finished. It makes it impossible to get into what you’re listening to. Once a tape is in the boombox it’s a hassle to put another in and wind it to the right spot, so the music is left alone. I don’t think I’m too malevolent of a music dictator, the only tape which seems to cause any consistent consternation is “Jazz hour (and a half)”…

I did look into playing music from an iPod as well. It has an analog input, but with a connector I haven’t seen before, described as “5 pin DIN”. I managed to find some specs on the pin layout and bought a cheapy 1/8″ jack to DIN cable. On first try I thought it wasn’t working, but when I turned the boombox all they way up I could hear music just faintly. I’m assuming that the DIN input expects a much higher line level than modern standards. The level is so low I’m sure some sort of preamp would safely boost the signal to a suitable amount. Some googling showed that as preamps are normally for audiophiles they are much too expensive, even basic headphone amplifiers. From there I read about the Chu Moy design of DIY headphone amplifier, but prebuilt kits are again, needlessly expensive. It looks like I could get the parts cheap enough, so it could be a future project…

In the end I settled on a cheap FM transmitter for my iPod. The sound quality can be iffy, depending on interference, but it’s good for a quick fix, and has the handy side effect of being able to control what’s playing without having to walk up to the boombox.

So all in all, I think I managed to put together a system that outperforms these modern iPod docks, and at a much more affordable price. It also has a huge amount of retro cool and gets looks and people asking about it. It’s also offered a great deal of fun, learning how to best use it, the history of the iconic boombox, and putting me more in touch with my favourite music and helping me share it with other people. It could be the start of a new hobby.