Category: USB stick

Followup roundup

Written by Fergal Hainey

I am thinking of reviving this blog. I am hearing from all angles why I should be publishing: in case what I share is useful, regular writing is supposed to be good for me, and because maybe “personal branding” is a fairly good idea after all.

The first thing I want to do is a draw a line under the old topics from 2012 while I was still at University of Aberdeen. I actually enjoyed reading them again, the topics are so unrelated to what I do now that they aren’t cringe worthy.

Boombox

My JVC ’80s boombox has had a good run. It has developed a hum on the tape deck that I can’t fix with my usual routine of poking the contacts with a spoon to ground them and twiddling the knobs to get static and dirt out. I don’t want to have to throw it away next time I move flat, so writing this has spurred me to contact a repair shop before I finished this paragraph.

In the end I used the FM transmitter, tape deck connector, and blank tapes pretty evenly depending on the circumstances. The FM transmitter if I needed to use my phone at the same time, the tape connector for better sound quality, and tapes for relaxing.

I have noticed an increased number of battery-powered small Bluetooth speakers that actually sound good. Finding something portable that sounded good without being too expensive is what drove me to buying a vintage boombox in the first place. The UE BOOM is the one I’ve heard the most of and I’m impressed by it. Some others sounded OK and there have been others that I didn’t think sounded good enough, but I didn’t take note of what they were. They seem to be in the £100 plus range still.

Node.js filtering proxy

I was really happy with how fast it is to get up and running with Node.js. I realise that forwarding all of your browser’s traffic isn’t a high load situation, but is more than Flask’s debug server would take with default settings for example. I appreciate that Node.js’ standard pattern worked so well out of the box, it’s evented and has a fast VM. Python is my first language now and I love it for how it encourages team maintainability, but often needs extra consideration to reach production performance. Python’s async syntax and PyPy should bridge the evented and VM gaps respectively. The new language features in ES6 should make JavaScript more attractive. Exciting times! I like to think of ease of maintenance by a team and performance as different facets of working at scale.

The code I wrote probably doesn’t work with 4OD anymore. I don’t have a television now so I don’t see adverts for shows, so I haven’t used any of the catch up services in years. It was a fun exercise though and gave me something to talk about in a job interview.

SD to HDD copier

I never tried it. I don’t hear the complaint of full SD cards so much anymore. I think the cloud and even lower SD card costs have solved the problem. I’m still excited about what new products these education focused electronics will lead to. The Arduino is still a great microcontroller. The Raspberry Pi is a great embeddable full computer and the new models are really powerful and still cheap. The new BBC micro:bit is awesome for experimenting with the components in modern mobile devices. I was worried that the micro:bit would be stepping on the Raspberry Pi’s toes, but after getting one at EuroPython 2016 I was won over. The micro:bit definitely has a focus on mobile devices, which makes a lot of sense as most people’s first computer is a mobile.

Universal repair USB stick

I feel guilty that I didn’t document my steps properly. I still have the one USB stick, but I’ve not had to use it in years. Modern Macs and Windows computers have built-in repair partitions. Upgrading to the latest OS is a lot more common now the upgrades are free and can be downloaded, so there seems to be fewer people with messed up systems. I myself hardly ever use a laptop anymore and do most of my work on a tablet. When a tablet goes bad, you just wipe it. That sounds like the worst case and evokes memories of reinstalling Windows. The reality is all the apps, settings, and documents are backed up on the cloud and starting from scratch again isn’t a big deal. With app stores for most platforms and Microsoft and Apple both peddling cloud storage, desktop could be going the same way. I’ve been considering trying to Dockerise my whole dev environment so even trickier installs are easier to reproduce. I’ve not seen anyone using Docker that way yet so could be interesting.

Clipboard sharing with Dropbox

I got a lot of use out of my little Dropbox app. Eventually messaging apps got better at accepting different sorts of pasted content and I stopped using it. I read the help page linked from my Public folder and it appears that only Pro and Business users can use Public folder simple URLs now, so my app may no longer work. If I were to make it now I’d probably have to use Dropbox’ API to make tokenised URLs. The help page also says that as of 2016-10-03 HTML documents won’t be rendered in a browser, I guess they’ll change the content type. My app relied on that for sharing rich text content.

The future

I do hope to keep this blog somewhat up to date now. Everyone in the tech industry has moments when they feel they were the first one to get a particular combination of parts working together, so I will try to share those times at least. I work mostly with Python, Docker, and AWS now so things could be more focused in that direction. That said I want to start making and computing as a hobby again not just work, I think about it so much.

Bootable USB stick followup

Written by Fergal Hainey

I discovered that using a GPT partition scheme meant that my stick wouldn’t work with older systems, which is a shame because I had a partition for using like a regular USB stick which is useful for transferring files around.

As it happens you can have disks that are both GPT and MBR, as detailed at http://www.rodsbooks.com/gdisk/hybrid.html (note that that guide is written by Rod Smith, the maintainer of rEFInd mentioned is my last post on this, and also gdisk, which I used for making hybrid MBRs before I realised it was also one of the things iPartition could do).

Another thought I had was: what about these news PCs that will be using UEFI (an updated and more standardised EFI)? I have some theories about being able to put the right file (say /efi/boot/bootx64.efi, which is a standard fallback location) in a partition by itself, with bootx64.efi being a GRUB EFI application made to use my existing HFS+ partition for it’s modules. Unfortunately I haven’t got any computers to test this on.

A bootable repair USB stick for Macs and PCs

Written by Fergal Hainey

Sounds useful. Should be a lot easier to do than it is. This has been an ongoing project of incremental improvements and sudden inspirations, I don’t think I’ll be able to document it all.

OK start from basics. To be bootable you need something to boot. GRUB is a bootloader that will work on PC and Mac. OK good, how to make a computer boot it. Using the age old method of BIOS booting. Put a little bit of machine code on the first block (512 bytes) of the disk, BIOS computers will run this, and it should do things like find any more machine code it needs and list and boot operating systems on the computer. GRUB needs to be installed to work on BIOS computers in sort of 3 parts. That first bit of boot code, which finds more boot code on the disk at a static offset which needs to be preprogrammed in the first part. Because there will be some sort of partitioning scheme like MBR that takes up the rest of the disk, the scheme will have to leave some space somewhere for the 2nd part of the machine code. Once GRUB has got all that together it should have enough functionality to access a volume/partition on the disk and load the rest of its modules and settings from the filesystem.

So I made an MBR partition scheme with enough enough free space on my new USB stick and set about installing grub on it. The grub distribution running on my Debian virtual machine was designed to tune the grub installation on the computer it was running on. I extracted elements from it to create a more generic set of grub installation scripts. And reading the manuals and doing some tests I managed to get an image of my USB stick to boot in a VM. Unfortunately I don’t have a spare computer lying around the test the actual USB stick.

GRUB has a nice bit of functionality which can make CD ISO9660 images (.iso) into loopbacked filesystems, and it is good at booting linux with just a kernel file and an initial RAM disk file. So any linux distribution which can be passed boot parameters to make itself aware of this and do a similar loopback mount of the iso (like SystemRescueCD and Debian install discs) can be booted like this without having to make separate partitions for them, which is nice.

A Windows install would be nice for fixing (read reinstalling) Windows stuff too. The one I have a license for is Windows 7 Professional 64 bit. I don’t think I’ll be able to do this with just an iso image. GRUB is not so good at booting windows, it relies on chainloading: being forwarded an address with more machine code to run to boot the operating system. I copied all the files on the windows install CD to a partition on the memory stick so most of the stuff it needs should be there. As far as I can tell there’s nothing on my Windows 7 install CD image to make the boot code for arbitrary partitions on a USB stick. Luckily I discovered ms-sys, which has recently added being able to to write what it calls an NTFS boot record, which I hoped I would be able to bootstrap the files on. Like the boot code at the start of a disk before the MBR partition table this boot record sits at the start of a partition, before the NTFS filesystem structure. Chainloading this boot record worked to my great surprise, I hadn’t found anyone else on the internet using this method.

So I had a USB stick that would theoretically work on a BIOS PC. Good start. Now lets’ looking into Macs. Macs don’t use BIOS. Bad start. They work with a sort custom version of an outdated EFI (Extensible Firmware Interface, designed to replace BIOS) spec. Firstly it required a GPT partition scheme to the disk. OK I can do that. Then there are two ways to specify what to boot, either by setting flags and details in the computer’s NVRAM, which require a system running which can do that (like the “bless” command on OS X) so is not ideal for making a repair disk, or with headers in Apple’s HFS+ filesystem. There are no other ways. So GRUB will have to use EFI, on a GPT partitioned disk with a HFS+ filesystem, very different to the BIOS, MBR, FAT32 setup I have.

Let’s see about reconciling the two. BIOS systems can work with GPT, there’s still the 1 block gap at the start of the disk. The GPT filesystem can then set aside some free space for the rest of GRUB. I can keep my old FAT32 partition with grub BIOS stuff in it. As long as there is a HFS+ partition with the right headers in it pointing to an EFI application, a Mac will be able to boot it. Grub can come in an EFI version, but it doesn’t do BIOS so I’ll need both versions on the stick. The Mac version will need its own modules, but should share the config file containing a list of operating systems and how to boot them with the BIOS version, GRUB config files have the “configfile” directive for this and grub can traverse different partitions and filesystem formats easily, as long as you include the right modules.

Around this time I learned that the Mac app I’ve had for a while, iPartition, was much more capable than I thought originally. It excels in nondestructive moving and resizing of partitions and changing the partition scheme and can give complete control over where on the disk partitions go. It could even easily create the BIOS boot partition GRUB needed on GPT disks. It gave me functionality I was expecting to need a collection of obscure command line tools for. I recommend it for anyone who needs to do more than basic disk partitioning.

At this point I had a USB stick that would boot in my bios based virtual machine, and would boot on my actual Mac. I consider that a success. I still can’t get Windows 7 install to work with EFI. You can’t chainload BIOS boot code from EFI. Another bootloader, rEFInd claims to be able to get into the Mac’s (and some other manufacturers’) CSM (Compatibility Support Module), but that may be cheating, and using two bootloaders would make things more complex. Windows 7 does support booting from EFI, and I’ve managed to make a little progress with the EFI app “bootmgfw.efi” on the install disk, but either it’s just doesn’t have the routines for the situation it’s in, or maybe I can change the BCD to make it work.