IBU

FreeBSD Remote Upgrades

http://www.berklix.com/~jhs/txt/host_upgrade.html

Author Julian H. Stacey

Purpose

Disclaimer: Total !

Matrix

----- 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3
----- 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
host1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
host2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
host3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
host4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Colour Scheme Documenting Action Numbers

Action Numbers

1. If this is an Internet live server, some days before, you have already changed timeouts for your domains, & patched out the IP of this box from your cluster. Assumptions: AMD running (if local secure net). Using Berklix standard AMD host mount path /host/hostname (not a FreeBSD default one). Shell is csh with a built in echo (not eg /bin/sh needing /bin/echo). echo $0 | grep csh Set pwd of root & toor & `whoami` (a normal login user) to known values. & grep `whoami` /etc/group | grep wheel You have already manually saved all user data & any source hacks, creating diffs on another system. echo "Do It Manually Now"

   
   

2. Backup /etc/ configs to other hosts in case of catastrophe cd / && rdist6 -P /usr/bin/ssh -f /etc/Distfile Backup to a spare Fdisk partition (for patch rescues) & also data to another host (for catastrophic rescue) & label dates & uname -r in /etc/fstab & /etc/motd

   
   

3. Make sure processor type of target can use what we have generated. dmesg | grep CPU:

   
   HOST=SOURCE
   

4. one can boot a cross installed boot kernel, where SOURCE host is built with cflags 686 & then make.conf gets "CFLAGS += -march=i586" before building for TARGET cpu=586, However thats Only true for kernel. Even with -march=i586 a 686 produces unusable binaries for eg init. grep CFLAGS /etc/make.conf Freebsd-7.4 + std gcc 4.2.1 fails to honour -march=i586 All these binaries were not executable on the 586: /boot/loader /sbin/init /bin/sh /bin/csh getty

   
   

5. Ensure we have all binaries & sources ready/ uploaded cd /usr/src && make obj cd /usr/src && make world Or cd /usr/src && make all

   
   

6. vi /usr/src/UPDATING Read & consider everything between old & new release, eg in the past it's been necessary to add a few passwd entries before make world would work (can't remember which, maybe some of _dhcp _pflogd _spamd _tss auditdistd bind daemon dlna haldaemon kmem messagebus proxy spamd unbound ) ; & periodicaly libc versions get increased, etc. To relate releases to dates see http://www.freebsd.org/releases/

   
   

7. Build GENERIC kernel, in case our specific kernel fails to configure or boot. Build host specific kernel. cd /sys/`uname -m`/conf && config GENERIC cd /sys/`uname -m`/compile/GENERIC make cleandepend && make depend && make setenv DESTDIR /pri/freebsd/releases/`uname -r`/586/destdir Orsetenv DESTDIR /pri/freebsd/releases/`uname -r`/686/destdirmake install cd /sys/modules ; make install cd /sys ; make install # recurses also into boot/ cd /sys/`uname -m`/compile/$TARGET.small && make depend && make

   
   

8. Build host specific kernel. cd /sys/`uname -m`/conf && config $TARGET.small cd /sys/`uname -m`/compile/$TARGET.small && make depend && make

   
   HOST=TARGET
   

9. Make space: Remove src ports doc obj rm /usr/src/.c* /usr/ports/.c* Cope with possible symbolic links of src ports doc obj" foreach i ( src ports doc obj ) echo "Emptying /usr/$i" ( rm -rf /usr/$i/* ) end

   
   

10. Check enough space free for new binaries (145M for 4.11, 332M for 7.4-RELEASE ) where /usr/destdir will move into file systems for / & /usr df / /usr* If not enough space try eg cd /usr && mv local /usr1/ && ln -s ../usr1/local /usr/local Either mkdir /usr/NEW.ALL Else (Which won't work so well, as moving across file systems when binaries are not in findable paths, & will fail & need manual intervention. mkdir /usr1/NEW.ALL ln -s -f ../usr1/NEW.ALL /usr/NEW.ALL

    
    HOST=SOURCE
    

11. Copy across (but do not install) binaries & sources before booting new kernel, (in case reboot fails & we have to do local login). . setenv target host setenv TARGET `echo $target | dd conv=ucase` setenv DESTDIR /host/$target/usr/NEW.ALL mkdir /host/$target/usr/NEW.ALL Can't do a direct install to root of target cos of lib problems over NFS so stage it via /usr/NEW.ALL cd /usr/src pushd etc make distrib-dirs make distribution make etc-examples make install popd make install cd /sys/`uname -m`/compile/GENERIC.small ; make install cd /sys/`uname -m`/compile/HOST.small ; make install HOST=TARGET Test Binaries (make sure eg you haven't installed 586 bins on a 486). (The test above is not adequate, it doesn't blow up if eg on a 486 target you run a 586 mount binary.) cd /usr/NEW.ALL/sbin && ./mount

    
    HOST=TARGET
    

12. Install src ports doc obj We did the binaries before this in case we ran out of space. cd /usr && mkdir src ports doc obj cd /usr && ls -al src ports doc obj cd /usr && chown -R -H jhs:staff src ports doc obj

    
    LOGIN=jhs
    

13. If $TARGET is allowed AMD mounts by $SOURCE: foreach i ( src ports doc ) cd /usr/$i && tar zxf /pri/FreeBSD/releases/7.4-RELEASE/tars/$i.tgz # Cannot yet use `uname -r` as target host is still running old release, # so type new release number manually. echo done $i end Else if $TARGET is a gate that $SOURCE does not allow AMD mounts from: HOST=SOURCE foreach i ( src ports doc ) cd /host/$target/usr/$i && tar zxf /pri/FreeBSD/releases/`uname -r`/tars/$i.tgz echo done $i end

    
    

14. cd /usr/ports && ln -s -f ../distfiles # Cannot yet use `uname -r` as target host is still running old release, # so type new release number manually. cd /usr/ports && ln -s -f ../packages && ln -s -f ../distfiles cd /usr/ports/.. && ln -s -f /pub/FreeBSD/dists/7.4-RELEASE distfiles cd /usr/ports/.. && ln -s -f /pub/FreeBSD/ports/`uname -m`/packages-`echo 7.4-RELEASE|dd conv=lcase` packages

    
    

15. chflags noschg /boot/kernel/kernel* cp /boot/kernel/kernel /kernel.OLD

    
    HOST=SOURCE
    

16. Optionally Install new live GENERIC kernel as fallback. Don't do this if eg Your network card is not configured for a GENERIC config. Your host is a HP Network ScanJet 5 & a Generic kernel will hang during boot on aic probe & never complete. setenv DESTDIR /host/$target/NEW cd /sys/`uname -m`/conf/../../compile/GENERIC && make install

    
    

17. Install new live custom kernel (old release or new release Generic becomes fallback) setenv DESTDIR /host/$target/NEW cd /sys/`uname -m`/conf/../../compile/$TARGET.small && make install

    
    HOST=TARGET
    

18. Do you have special /boot/loader.conf /boot/kernel.conf that might be essential to boot. find /boot -name \*.conf | grep -v /defaults/

    
    HOST=SOURCE
    

19. Install new boot & kernel modules setenv DESTDIR /host/$target cd /sys && make install

    
    HOST=TARGET
    

20. cd /etc/mail ; make stop vi -c/sendmail_enable /etc/rc.conf # sendmail_enable="NO" to avoid start on after reboot, before new configs in place cd /usr/local/etc/rc.d ; ./apache stop /etc/rc.d/named stop # kill -9 `cat /var/run/named/pid` ps -laxxww # look for anything else to kill echo "Upgrading now, kicking users off" | wall rsh local fetchmail # Drain mail

    
    

21. Last chance to Think ! rm -f /var/run/nologin /etc/nologin # (Path in /etc/login.conf & man sshd ) Ensure no top directories lack permissions (eg from inherited umask from mv / cp) else eg if /bin is drwx------ you will get /bin/csh: Permission denied cd / ; ls -l | grep rwx--- Ensure alternate rescue entries. grep "^PermitRootLogin" /etc/ssh/sshd_config Look for "yes", else if normal user login fails, you'll be stuck. grep toor /etc/passwd ; vipw Ensure rlogin & telnet also allow login, because after upgrading from 6.1 to 6.3 I got: ssh_exchange_identification: Connection closed by remote host mv /etc/pam.conf /etc/pam.conf.MV Ensure no obsolete /etc/pam.conf might lock you out. Optional sanity check before upgrade of binaries, Consider if the day & time of day allows human presence at remote site in case someone has to push reset &/or more drastc rescue, then: reboot

    
    

22. Move new binary trees to right file systems. mkdir /NEW /OLD /usr/OLD /var/OLD mv /usr/NEW.ALL/var /var/NEW mv /usr/NEW.ALL/usr /usr/NEW Next line complex to avoid just moving a possible symbolic link You can't just do this cd /usr/NEW.ALL/../ && mv NEW.ALL /NEW because /rescue contains 121 sym links of 4 Meg, so do this: chown -R -noschg /usr/NEW.ALL ( cd /usr/NEW.ALL && tar cf - . ) | ( cd /NEW && tar xf - ) cd /usr/NEW.ALL && find . -type f -exec cmpd -d {} /NEW \;

    
    

23. Installing /var cd /var/NEW && foreach i ( * ) echo "Starting $i" mv /var/$i /var/OLD/$i mv $i /var/$i echo "Finished $i" end rm -rf /var/NEW/*

    
    

24. Installing /usr cd /usr/NEW && foreach i ( * ) echo "Starting $i" mv /usr/$i /usr/OLD/$i mv $i /usr/$i echo "Finished $i" end

    
    

25. Installing Root find /boot -name \*.conf | grep -v /defaults/ ( cp /boot/loader.conf /NEW/boot/ ) ( cp /boot/kernel.conf /NEW/boot/ ) rm /NEW/sys foreach i ( .cshrc .profile dev etc root ) mv /NEW/$i /$i.NEW end Copy tools to known location, normally only mv is needed, but if you get anything wrong, eg on the wrong file system, mv will fail to fork /bin/cp ( cd /bin && file cp ls mv pwd rm | grep "statically linked" ) Ensure they are static, not dynamic. cd /bin && cp cp ls mv pwd rm / ) cd /NEW && foreach i ( * ) echo "Starting $i" /mv /$i /OLD/$i /mv /NEW/$i /$i echo "Finished $i" end (cd / && rm mv cp rm pwd ls ) rm /etc/objformat /usr/bin/objformat cmp /usr/src/etc/rc /etc/rc # See if mergemaster has failed to update, (when that happened next reboot had no net & hostname Amnesiac)

    
    

26. Test if system is bootable, & load new shared libraries etc. reboot

    
    

27. Delete old binaries. rmdir /NEW /usr/NEW cd / && ls /OLD /usr/OLD chflags -R noschg /OLD /usr/OLD /kernel.Old rm -rf /mv /OLD /usr/OLD /kernel.Old ls /var/NEW echo "Do It Manually Now"

    
    

28. Optionally fix some rc.conf & rc.local type errors, (visible after booting with X off). But some errors not worth fixing as we haven't upgraded /etc/ scripts yet. echo "Do It Manually Now"

    
    

29. Manually also install /usr/local if a major release upgrade, where old shared libs wont work any more. echo "Do It Manually Now"

    
    LOGIN=jhs

30. cd tmp && script Optionally, don't customise if anything might break world, eg SASL on a gate cd /usr/src && customise `pwd` (Or better customise ports too with a simpler more complete:) cd ~/tmp script customise

    
    

31. cd /usr/src && make obj

    
    LOGIN=root

32. Rebuild binaries with local /etc/make.conf CFLAGS cd /usr/src sh nohup make world & (nohup allows build (running for hours) to continue, even if you later reboot or halt or drop the link of the controlling screen. At any time monitoring is resumable with tail -f /host/$target/usr/src/nohup.out

    
    

33. Compare & upgrade /etc with/from /usr/src/etc cd /etc && tar zcf ../etc.`date +%Y_%m_%d_%H_%M_%S`.tgz . mergemaster -sicv

    
    

34. You need a new /etc/mail/sendmail.cf by now else you wont get error logs mailed cd /usr/obj/`cd /usr/src;/bin/pwd`/etc/sendmail; cp `hostname`.cf /etc/mail cp *submit* /etc/mail cd /etc/mail && csh

    
    

35. Optionally reinstall packages, or mandatory if a major release upgrade. cd /var/db/pkg && ls pkg_info -a You may want to upgrade with some of /usr/ports/*/pkg* rlogin #source ; cd /host/$target/usr/local (cd /usr/local && tar cf - . ) | tar xf - rlogin #source ; cd /host/$target/var/db/pkg ( cd /var/db/pkg && tar cf - . ) | tar xf -

    
    

36. Remake Ports: Optional on minor numbers, mandatory on major releases. portsupgrade Or cd /usr/ports ; nice make -k BERKLIX_CLIENT=YES BERKLIX_GATE=YES BERKLIX_MINIMAL=YES package Warning: FreeBSD is now in the habit of periodicaly discarding ports, so some won't rebuild & binaries may be old.

    
    

37. Detect binaries that failed to rebuild. foreach i ( /*bin /usr/*bin /usr/local/bin ) echo Listing $i ls -ltr $i echo " " end

    
    

38. Look to see errors in /var/log/, especially messages. cd /var/log ; more messages

    
    

39. If this is a live internet server, check: ftpd: Ensure no incoming etc. popd Enable before smtpd smtp Enable before named httpd Enable & check more /site/domain/this/usr/local/etc/apache/httpd.conf named: re-enable the IP into domain cluster, & reset DNS timeouts to longer defaults. cd /site/domain/berklix.org/etc/namedb ; vi master.fwd ; make ; more /site/Distfile /site/domain/init ssh: Check ssh allows login, then Optionally: vi /etc/inetd.conf # comment out rlogin telnet kill -HUP `cat /var/run/inetd.pid` # To make machine safer, (unless all users are aware rlogin & telnet are dangerous, can be packet sniffed to steal passwords, so normal users should use ssh.) SASL: grep CFLAGS /etc/make.conf | grep -i sasl grep .include /etc/make.conf cd /usr/src ; make world customise make ; make install Upgrade /etc/mail/sendmail.cf & re-enable