Skrevet av Emne: Copying large amounts of small files using SMB/CIFS  (Lest 4495 ganger)

Utlogget Floyd-ATC

  • Livstidsdiktator
  • Administrator
  • Guru
  • *****
  • Innlegg: 542
  • Karma: +12/-0
    • MSN Messenger - floyd@atc.no
    • Vis profil
    • floyd.atc.no
    • E-post
Copying large amounts of small files using SMB/CIFS
« på: 17. April 2015, 14:41 pm »
  • [applaud]0
  • [smite]0
  • We encountered a scenario where more than 500000 small files needed to be copied from a CIFS share running on a NetApp to a local USB disk. To make matters even more interesting, the NetApp uses on-access virus scanning. I should point out that we wanted to discard file times, ownership and permissions if possible. The USB disk was therefore formatted as exFAT rather than NTFS.

    This was solved using a laptop computer with an USB3 port and a Gigabit ethernet port.

    First, a plain old Ctrl+C/Ctrl+V file copy achieved a peak throughput of about 3 Mbytes/sec, which would complete the copying of 530 Gbytes in just over 2 days. TeraCopy yielded the exact same throughput. This was completely unacceptable.

    Second, we downloaded RichCopy, a tool originally developed by Microsoft for internal use but later released to the public. Using 50 paralell threads and request serialization we achieved a peak throughput of nearly 8 Mbytes/sec, reducing the estimated time needed to under 23 hours.

    Third, we booted a Knoppix live DVD, mounted the exFAT USB drive and CIFS share and used the following commands:
    Kode: [Velg]
    time find Source -noleaf -type d | xargs -n 1 -P 10 -I % mkdir -p /mnt/destination/%
    Kode: [Velg]
    time find Source -noleaf -type f | xargs -n 1 -P 50 -I % cp % /mnt/destination/%
    Here, "Source" is the relative name of the directory containing the files to be copied.
    "/mnt/destination" is the USB drive mount point.
    The first command is used to build the directory structure using up to 10 paralell processes. This took about 10 minutes.
    The second command is used to copy the actual files using up to 50 paralell processes. This took just over 7 hours with an average throughput of just over 20 Mbytes/sec.
    Note the "-I %" which means that the character "%" should be substituted with the actual directory or file name to be created/copied.

    It is possible to have 'cp' keep file times using the appropriate options, note however that this may affect the performance. You should also experiment with the number of paralell processes to find the "sweet spot" for your particular scenario. Not all USB drives, computers, network switches and CIFS servers are the same. That said, the most important factor is probably the number and size of the files to be copied.

    I should also mention that the GUI on our Knoppix 7.4.2 crashed halfway through one of the tests, possibly due to a resource leak in the window manager or display driver. This did not seem to affect the performance but we decided to boot using the CLI only by typing "knoppix 2" at the bootloader prompt. As a bonus, this freed up some system resources, improving the performance even more.  :)


    -Floyd.

    --
    Det finnes 10 typer mennesker;
    de som forstår binærtall, de som ikke gjør det, og de som forstår Grey code.

    Utlogget UnknownStranger

    • n00b
    • *
    • Innlegg: 12
    • Karma: +0/-1
      • Vis profil
      • E-post
    Sv: Copying large amounts of small files using SMB/CIFS
    « Svar #1 på: 01. November 2023, 12:40 pm »
  • [applaud]0
  • [smite]0
  • Colored diamond jewelry is a treasure chest of emotions. These gems mirror the full spectrum of feelings, from the passion of a coloring diamonds red diamond to the tranquility of a green one. Wearing colored diamond jewelry is like carrying a palette of emotions with you.



    Utlogget UnknownStranger

    • n00b
    • *
    • Innlegg: 12
    • Karma: +0/-1
      • Vis profil
      • E-post
    Sv: Copying large amounts of small files using SMB/CIFS
    « Svar #2 på: 01. November 2023, 13:08 pm »
  • [applaud]0
  • [smite]0
  • Supporting local artists by wearing their designs on your t-shirt helps promote the arts in your community. Custom t-shirt printing Screen printing Services allows fitness enthusiasts to create their workout gear. You can design shirts that motivate you during exercise or display your fitness achievements.



    Utlogget UnknownStranger

    • n00b
    • *
    • Innlegg: 12
    • Karma: +0/-1
      • Vis profil
      • E-post
    Sv: Copying large amounts of small files using SMB/CIFS
    « Svar #3 på: 01. November 2023, 13:25 pm »
  • [applaud]0
  • [smite]0
  • Making your laptop more user-friendly wireless earbuds or headphones with excellent sound quality are a must for enjoying your smartphone accessories favorite music or podcasts on the go. A laptop mount or arm allows you to free up desk space by mounting your laptop on an adjustable arm.



    Utlogget UnknownStranger

    • n00b
    • *
    • Innlegg: 12
    • Karma: +0/-1
      • Vis profil
      • E-post
    Sv: Copying large amounts of small files using SMB/CIFS
    « Svar #4 på: 01. November 2023, 13:35 pm »
  • [applaud]0
  • [smite]0
  • I was pleasantly surprised by how much fun the New Year's Eve murder mystery party turned out to be. The attention to detail was incredible how to plan a murder mystery dinner party and the immersive ambiance added an extra layer of authenticity to the experience.



    Utlogget Berlin

    • n00b
    • *
    • Innlegg: 19
    • Karma: +0/-0
      • Vis profil
    Sv: Copying large amounts of small files using SMB/CIFS
    « Svar #5 på: 02. November 2023, 07:27 am »
  • [applaud]0
  • [smite]0
  • Most of the trunks and elements are fit for the options. The phase of the what is halotherapy for the funds. Style is implied for the challenge. Push up is held for the terms for thk approval of the dynamics. Schleps welcomed for the mild for the assessment for the options for all trunks.



    Utlogget Berlin

    • n00b
    • *
    • Innlegg: 19
    • Karma: +0/-0
      • Vis profil
    Sv: Copying large amounts of small files using SMB/CIFS
    « Svar #6 på: 02. November 2023, 07:44 am »
  • [applaud]0
  • [smite]0
  • Mobility scooters and related equipment are not limited to outdoor use alone. Many models are designed with maneuverability in mind folding power wheelchair making them equally suitable for indoor navigation.