Building Elder Signs Cluster - Part 1

The continued effort to turn six Odroid-MC1 Solos into a PXE booted computer cluster on which to engage in further FOSS hijinks. The last post documented the physical set up of the devices as well as the creation and basic configuration of an image on a MicroSD card that was capable of booting the device into a state which allowed remote connection to an MC1 node via ssh.

In this post I hope to document the process of creating a minimal MicroSD card image capable of booting the MC1 nodes into the same state via Preboot Execution Environment (PXE) from Network-attached storage (NAS), using Trivial File Transfer Protocol (TFTP) to boot and Network File System (NFS) to provide the root file systems.

Overview

Network Setup

The work to set up and configure the MicroSD cards as PXE clients will be done on the Setback machine. Boot files and file systems will be served from the Monolith device.

Assumptions (Will need to be modified to the specifics of your environment)
- MicroSD cards, when inserted on Setback are: /dev/sdc
- MicroSD cards, when mounted for file system access will be at: /mnt/sd
- PXE will use NAS IP address: 192.168.1.89
- NAS share for TFTP boot files is: /data/tftpboot
- NAS share for NFS file systems is: /data/tftproot

MicroSD Setup PXE Client

Format MicroSD Card

Insert MicroSD card into reader on Setback to show up as /dev/sdc
- The SD card, should be new or contain data no longer of interest as anything on the card will be lost as part of the partitioning and file system formatting.
- Be sure you are really working on the device you want to be working on as data will be lost on any device targeted as part of this process.
MicroSD card will need a FAT partition to contain the boot.ini file required for the MC1 devices. There are many ways to remove and create partitions from disk devices. I am using fdisk in this post, but any tool you are comfortable with may be used instead.
32GB MicroSD card before partitioning.

chuck@setback ~ $ lsblk /dev/sdc
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc      8:32   1  30G  0 disk
└─sdc1   8:33   1  30G  0 part

chuck@setback ~ $ fdisk -l /dev/sdc
Disk /dev/sdc: 30.1 GiB, 32220119040 bytes, 62929920 sectors
Disk model: SD/MMC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x134de435

Device     Boot Start      End  Sectors Size Id Type
/dev/sdc1  *       64 62929919 62929856  30G  c W95 FAT32 (LBA)
chuck@setback ~ $

Partitioning the MicroSD card using fdisk. The important part in the below is the initial 128 MiB partition created, as that will be home to the boot.ini file and is the only partition related in any way to the PXE boot process. I just didn’t want to let the rest of the disk go unused and it will probably turn out to be helpful to have storage local to each node. With fdisk, I delete existing partition, create a new 128 MiB partition and then create another partition that uses the rest of the disk.

chuck@setback ~ $ sudo fdisk /dev/sdc

Welcome to fdisk (util-linux 2.35).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-62929919, default 2048): 2048
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-62929919, default 62929919): 264191

Created a new partition 1 of type 'Linux' and of size 128 MiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): c
Changed type of partition 'Linux' to 'W95 FAT32 (LBA)'.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (264192-62929919, default 264192): 264192
Last sector, +/-sectors or +/-size{K,M,G,T,P} (264192-62929919, default 62929919): 62929919

Created a new partition 2 of type 'Linux' and of size 29.9 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

chuck@setback ~ $

Create file systems on MicroSD card partitions.

chuck@setback ~ $ sudo mkfs.vfat -n boot /dev/sdc1
mkfs.fat 4.1 (2017-01-24)
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
chuck@setback ~ $ sudo mkfs.ext3 -n l_data /dev/sdc2
mke2fs 1.45.5 (07-Jan-2020)
chuck@setback ~ $

MicroSD after partitioning and formating

chuck@setback ~ $ lsblk /dev/sdc
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdc      8:32   1   30G  0 disk
├─sdc1   8:33   1  128M  0 part
└─sdc2   8:34   1 29.9G  0 part

chuck@setback ~ $ fdisk -l /dev/sdc
Disk /dev/sdc: 30.1 GiB, 32220119040 bytes, 62929920 sectors
Disk model: SD/MMC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x134de435

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sdc1         2048   264191   262144  128M  c W95 FAT32 (LBA)
/dev/sdc2       264192 62926830 62662639 29.9G 83 Linux

Finalize Micro SD Card

Mount MicroSD partition and copy boot.ini to it.
- Remove and reinsert MicroSD card.
- I do not allow removable media to auto mount, so I need to mount partition manually.

chuck@setback ~ $ sudo mount /dev/sdc1 /mnt/sd

chuck@setback ~ $ sudo wget -O /mnt/sd/boot.ini https://git.io/v5ZNg
--2020-02-02 05:53:17--  https://git.io/v5ZNg
Resolving git.io... 52.206.168.246, 54.164.7.157, 34.204.59.252, ...
Connecting to git.io|52.206.168.246|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/mdrjr/5422_bootini/master/boot.ini [following]
--2020-02-02 05:53:17--  https://raw.githubusercontent.com/mdrjr/5422_bootini/master/boot.ini
Resolving raw.githubusercontent.com... 151.101.128.133, 151.101.192.133, 151.101.0.133, ...
Connecting to raw.githubusercontent.com|151.101.128.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11003 (11K) [text/plain]
Saving to: ‘/mnt/sd/boot.ini’

/mnt/sd/boot.ini    100%[=================================================>]  10.75K  --.-KB/s    in 0s      

2020-02-02 05:53:17 (36.9 MB/s) - ‘/mnt/sd/boot.ini’ saved [11003/11003]

chuck@setback ~ $

Edit boot.ini on MicroSD card.
- Using text editor of your choice, modify the boot.ini file on the MicroSD card.
  - Comment out the current last line of file: # bootz 0x40008000 0x42000000 0x44000000
  - Add line to end: setenv serverip 192.168.1.89
  - Add line to end: setenv bootfile default-arm-exynos
  - Add line to end: run bootcmd_pxe

sudo vim /mnt/sd/boot.ini

Get and fuse u-boot to SD card.
- Change to a working directory in order to clone u-boot project
- Apply u-boot to the MicroSD card

chuck@setback /mnt/sd $ cd /mnt/monolith/projects/MC1
chuck@setback /mnt/monolith/projects/MC1 $ git clone --depth 1 https://github.com/hardkernel/u-boot.git -b odroidxu4-v2017.05
Cloning into 'u-boot'...
remote: Enumerating objects: 13915, done.
remote: Counting objects: 100% (13915/13915), done.
remote: Compressing objects: 100% (12554/12554), done.
remote: Total 13915 (delta 2654), reused 5304 (delta 1119), pack-reused 0
Receiving objects: 100% (13915/13915), 17.53 MiB | 11.95 MiB/s, done.
Resolving deltas: 100% (2654/2654), done.
Updating files: 100% (12705/12705), done.
chuck@setback /mnt/monolith/projects/MC1 $ cd u-boot/sd_fuse
chuck@setback /mnt/monolith/projects/MC1/u-boot/sd_fuse $ sudo ./sd_fusing.sh /dev/sdc
+ '[' -z /dev/sdc ']'
+ '[' -b /dev/sdc ']'
+ echo '/dev/sdc reader is identified.'
/dev/sdc reader is identified.
+ '[' -d /sys/block/sdcboot0 ']'
+ '[' -n '' ']'
+ signed_bl1_position=1
+ bl2_position=31
+ uboot_position=63
+ tzsw_position=1503
+ device=/dev/sdc
+ env_position=2015
+ '[' -f ./u-boot-dtb.bin ']'
+ '[' -f ./u-boot.bin ']'
+ '[' -f ../u-boot-dtb.bin ']'
+ '[' -f ./u-boot.bin.hardkernel ']'
+ uboot=./u-boot.bin.hardkernel
+ echo 'BL1 fusing'
BL1 fusing
+ sudo dd iflag=dsync oflag=dsync if=./bl1.bin.hardkernel of=/dev/sdc seek=1
30+1 records in
30+1 records out
15616 bytes (16 kB, 15 KiB) copied, 0.0410045 s, 381 kB/s
+ echo 'BL2 fusing'
BL2 fusing
+ sudo dd iflag=dsync oflag=dsync if=./bl2.bin.hardkernel.720k_uboot of=/dev/sdc seek=31
28+1 records in
28+1 records out
14592 bytes (15 kB, 14 KiB) copied, 0.0749771 s, 195 kB/s
+ echo 'u-boot fusing'
u-boot fusing
+ sudo dd iflag=dsync oflag=dsync if=./u-boot.bin.hardkernel of=/dev/sdc seek=63
1211+1 records in
1211+1 records out
620062 bytes (620 kB, 606 KiB) copied, 1.42512 s, 435 kB/s
+ echo 'TrustZone S/W fusing'
TrustZone S/W fusing
+ sudo dd iflag=dsync oflag=dsync if=./tzsw.bin.hardkernel of=/dev/sdc seek=1503
512+0 records in
512+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.827659 s, 317 kB/s
+ echo 'u-boot env erase...'
u-boot env erase...
+ sudo dd iflag=dsync oflag=dsync if=/dev/zero of=/dev/sdc seek=2015 bs=512 count=32
32+0 records in
32+0 records out
16384 bytes (16 kB, 16 KiB) copied, 0.0776771 s, 211 kB/s
+ echo 'U-boot image is fused successfully.'
U-boot image is fused successfully.
+ echo 'Eject /dev/sdc and insert it again.'
Eject /dev/sdc and insert it again.
chuck@setback /mnt/monolith/projects/MC1/u-boot/sd_fuse $ sudo umount /mnt/sd
chuck@setback /mnt/monolith/projects/MC1/u-boot/sd_fuse $

Next Steps

Take the MicroSD card and put it into an MC1 node and see if it works.
Console over UART serial port is essential for trouble shooting.