
    <h2>
      Transferring Large Data Sets</h2>
    <p>
      Very large data sets present their own problems. Not everyone has
      directories with hundreds of gigabytes of project files, but I do,
      and I assume I'm not the only one.</p>
    <p>
      For instance, I have a directory with over 700 radio shows, many
      of these directories also have a podcast, and they also have
      pictures and text files.</p>
    <p>
      Doing a properties check on the directory I see 450 gigabytes of
      data.</p>
    <p>
      When I started envisioning Libre Indie Archive I wanted to move
      the directories into archival storage using optical drives. My
      first attempt at this didn't work because I lost metadata when I
      wrote the optical drives since optical drives are read only.</p>
    <p>
      After further work and study I learned that tar files can preserve
      meta data if they are created and uncompressed as root. In fact,
      if you are running tar as root preserving file ownership and
      permissions is the default.</p>
    <p>
      So this means that optical drives are an option if you write tar
      archives onto the optical drives.</p>
    <p>
      I have better success rates with 25 GB Blue Ray Discs than with
      the 50 GB discs. So, if your directory breaks up into projects
      that fit on 25 GB discs, that's great.</p>
    <p>
      My data did not do this easily but tar does have an option to
      write a data set to multiple tar files each with a maximum size,
      labelling them -0 -1, etc. </p>
    <p>
      When using this multi volume feature you cannot use compression.
      So you will get tar files, not tar.gz files.</p>
    <p>
      It's better to break the file sets up in more reasonable sizes so
      I decided to divide the shows up alphabetically by title, so all
      the shows starting with the letter a would be one data set and
      then down the alphabet, one letter at a time.</p>
    <p>
      Most of the letters would result in a single tar file labeled -0
      that would fit on the 25 GB disc. Many letters, however, took two
      or even three tar files that would have to be written on different
      disks and then concatenated on the primary system before they are
      extracted to the correct location in primaryfiles.</p>
    <p>
      There is a companion program to tar, called tarcat, that I used to
      combine 2 or 3 tar files split by length into a single tar file
      that could be extracted.</p>
    <p>
      I ran engrampa as root to extract the files.</p>
    <p>
      So, I used a tar command on the working system where my Something
      Blue radio shows are stored. Then I used K3b to burn these files
      onto a 25 GB Blu Ray Disc carefully labeling the discs and writing
      a text file that I used to keep up with which files I had already
      copied to Disc.</p>
    <p>
      Then on the Libre Indie Archive primary system I copied from the
      Blu Ray to the boot drive the file or files for that data set.
      Then I would use tarcat to combine the files if there was more
      than one file for that data set. And finally I would extract the
      files to primaryfiles by running engrampa as root.</p>
    <p>
      Now I'm going to go into details on each of these steps.</p>
    <p>
      First make sure that the Libre Indie Archive program, prep.sh, is
      in your home directory on your workstation. Then from the data
      directory to be archived, in my case the something_blue directory
      run prep.sh like this.</p>
    <pre>
~/prep.sh</pre>
    <p>
      This will create a file named IA_Origin.txt that lists the date,
      the computer and directory being archived, and the users and
      userids on that system. All very helpful information to have if at
      some time in the future you need to do a restore.</p>
    <p>
      Next create a tar data set for each letter of the alphabet. (You
      may want to divide your data set in a different way.)</p>
    <p>
      Open a terminal in the same directory as the data directory, my
      something_blue directory, so that ls displays something_blue (your
      data directory). I keep the Something Blue shows and podcasts in
      subdirectories in the something_blue directory.</p>
    <p>
      Here's the tar command.</p>
    <p>
      Example a:</p>
    <pre>
sudo tar -cv --tape-length=20000000 --file=somethingblue-a-{0..50}.tar /home/larry/delta/something_blue/a*</pre>
    <p>
      This is for the letter a so the --file parameter includes the
      letter a. The numbers 0..50 in the squirelly brackets are the
      sequence numbers for the files. I only had one file for the letter
      a, somethingblue-a-0.tar.</p>
    <p>
      The last parameter is the source for the tar files, in this case</p>
    <pre>
/home/larry/delta/something_blue/a*</pre>
    <p>
      All of the files and directories in the something_blue directory
      that start with the letter a.</p>
    <p>
      You may want to change the --tape-length parameter. As listed it
      stores up to 19.1 GB. The maximum capacity of a 25 GB Blu-ray is
      23.3GB for data storage. </p>
    <p>
      Example b:</p>
    <p>
      For the letter b, I ended up with three tar files.</p>
    <pre>somethingblue-b-0.tar<br>somethingblue-b-1.tar<br>somethingblue-b-2.tar</pre>
    <p>
      I will use these files in the example below using tarcat to
      combine the files.</p>
    <p>
      I use K3b to burn Blu-Ray data discs. Besides installing K3b you
      have to install some other programs and then there is a particular
      setup that needs to be done including selecting cdrecord and no
      multisession. Here's an excellent article that will go step by
      step&nbsp;through the installation and setup.</p>
    <p>
      How to burn Blu-ray discs on Ubuntu and derivatives using K3b?</p>
    <p>
      <a moz-do-not-send="true"
href="https://en.ubunlog.com/how-to-burn-blu-ray-discs-on-ubuntu-and-derivatives-using-k3b/">https://en.ubunlog.com/how-to-burn-blu-ray-discs-on-ubuntu-and-derivatives-using-k3b/</a></p>
    <p>
      I also always check Verify data and I use the Linux/Unix file
      system, not windows which will rename your files if the filenames
      are too long.</p>
    <p>
      I installed a Blu-Ray reader into the primary system and I used
      thunar to copy the files from the Blu-Ray Disc to the boot drive.</p>
    <p>
      In the primaryfiles directory I make a subdirectory,
      something_blue, to hold the archived shows.</p>
    <p>
      If there is only one file, like in example a above, you can skip
      the concatenation step.</p>
    <p>
      If there is more than one file, like Example b above, you use
      tarcat to concatenate these files into one tar file.</p>
    <p>
      You have to do this. If you try to extract from just one of the
      numbered files when there is more than one you will get an error.
      So if I try to extract from somethingblue-b-0.tar and I get an
      error it doesn't mean that there's anything wrong with that file.
      It just has to be concatenated with the other b files before it
      can be extracted.</p>
    <p>
      There is a companion program to tar called tarcat that should be
      used to concatenate the tar files.</p>
    <p>
      Here's the command I used for example b, above.</p>
    <pre>
tarcat somethingblue-b-0.tar somethingblue-b-1.tar somethingblue-b-2.tar &gt; sb-b.tar</pre>
    <p>
      This will concatenate the three smaller tar files into one bigger
      tar file named sb-b.tar</p>
    <p>
      In order to preserve the meta data you have to extract the files
      as root. In order to make it easier to select the files to be
      extracted and where to store them I use the GUI archive manager,
      engrampa. To run engrampa as root open a terminal with CTRL-ALT t
      and use this command</p>
    <pre>sudo -H engrampa</pre>
    <p>
      Click Open and select the tar file to extract. Then follow the
      path until you are in the something_blue directory and you are
      seeing the folders and files you want to extract. Type Ctrl a to
      select them all. (instead of the something_blue directory you will
      go to your_data directory)</p>
    <p>
      Then click Extract at the top of the window. Open the directory
      where you want the files to go. In my case,
      primaryfiles/something_blue</p>
    <p>
      Then click Extract again in the lower right.</p>
    <p>
      After the files are extracted go to your data directory in
      primaryfiles and check that the directories and files are where
      you expect them to be.</p>
    <p>
      You can also open a terminal in that directory and type</p>
    <pre>
ls -l</pre>
    <p>
      to review the meta data.</p>
    When dealing with data chunks sized 20 GB or more each one of these
    steps takes time. The reason I like using an optical disk backup to
    transfer the files from the working system to Libre Indie Archive is
    because it gives me an easy to store backup that is not on a
    spinning drive and that cannot be overwritten. Still optical disk
    storage is not perfect either. It's just another belt to go with
    your suspenders.
    <p>
      Another way to transfer directories into the primaryfiles
      directory is with ssh over the network. This is not as safe as
      using optical disks and it also does not provide the extra
      snapshot backup. It also takes a long time but it is not as labor
      intensive.</p>
    <p>
      After I spend some more time thinking about this and testing I
      will do a podcast about transferring large data sets with ssh.</p>
    <p>
      Although I am transferring large data sets to move them into
      archival storage using Libre Indie Archive there are many other
      situations where you might want to move a large data set while
      preserving the meta data. So what I have written about tar files,
      optical discs, and running thunar and engrampa as root is
      generally applicable.</p>
    <p>
      As always comments are appreciated. You can comment on Hacker
      Public Radio or on Mastodon. Visit my blog at home.gamerplus.org
      where I will post the show notes and embed the Mastodon thread for
      comments about thie podcast.</p>
    <p>
      Thanks</p>
    <p>
      <br>
    </p>
    <p>
      <br>
    </p>