Well I did two copies of the DPX folders for the two shows to brand new 4TB raw drives and the initial prompt was 6 hours for both shows and it did seem to run approximately that fast. I agree with your logic but my experience hasn't shown that the brand new single raw drive is half the speed of the Graids but something in the range of 25% slower. Perhaps some of the Graids had more fragmentation on them but the two 20TB Graids were brand new.
Also given the large size of the DPX files perhaps the difference in show lengths is more appreciable as far as overall data size being transferred. Could the finder also get mislead when it starts the copy starting I assume with the outer tracks which are moving faster and can store data more quickly at the beginning slow up enough to make the finder re access the duration needed to copy?
I'll pay more attention to the details in the future but I can assure you none of the files too anywhere near 10 to 14 hours going to the raw drive.
---In Avid-L2@yahoogroups.com, <bogdan_grigorescu@...> wrote :
point is, if the 20TB drive , which is a 2way stripe(RAID0) reports 5-7 hours, then the single spindle would probably take 10-14 hours to copy the same amount. No?
BG
www.finale.tv
From: "bigfish@... [Avid-L2]" <Avid-L2@yahoogroups.com>
To: Avid-L2@yahoogroups.com
Sent: Thursday, December 15, 2016 8:39 PM
Subject: Re: [Avid-L2] OT: Shotput Pro Copy/Process times?
The single raw drive is a Hitachi NAS Deskstar 4TB and I think you are probably more accurate I might have confused the 5 hour prompt with a 6 hour prompt from last week now that I think about it.
As far as the Graid it's a 20TB drive. Yesterday the exact same model with virtually the same amount of data on it prompted 5 hours for the DPX copy where as tonight for a similar situation it says 7 hours. I can only guess that must have something to do with the Raid Tower Source being more fragmented on the source folder given it was the last DPX sequence exported to the Raid Tower and I had to delete other folders to make room I would assume it is less contiguous than the other files. The actual source folders are roughly the same size with the shows durations being within a couple minutes of each other.
---In Avid-L2@yahoogroups.com, <bogdan_grigorescu@...> wrote :
From: "bigfish@... [Avid-L2]" <Avid-L2@yahoogroups.com>
To: Avid-L2@yahoogroups.com
Sent: Thursday, December 15, 2016 7:10 PM
Subject: Re: [Avid-L2] OT: Shotput Pro Copy/Process times?
---In Avid-L2@yahoogroups.com, <bigfish@...> wrote :
As far as the Graid it's a 20TB drive. Yesterday the exact same model with virtually the same amount of data on it prompted 5 hours for the DPX copy where as tonight for a similar situation it says 7 hours. I can only guess that must have something to do with the Raid Tower Source being more fragmented on the source folder given it was the last DPX sequence exported to the Raid Tower and I had to delete other folders to make room I would assume it is less contiguous than the other files. The actual source folders are roughly the same size with the shows durations being within a couple minutes of each other.
---In Avid-L2@yahoogroups.com, <bogdan_grigorescu@...> wrote :
what's the exact model of the destination drive?
BG
www.finale.tv
From: "bigfish@... [Avid-L2]" <Avid-L2@yahoogroups.com>
To: Avid-L2@yahoogroups.com
Sent: Thursday, December 15, 2016 7:10 PM
Subject: Re: [Avid-L2] OT: Shotput Pro Copy/Process times?
As a follow up I just started a similar copy from the same Raid Tower to a Graid brand new 20 TB that I already exported a similar sized DPX sequence today from Avid. Now I'm doing a finder level copy from the Raid tower to this other than the aforemention DPX export is empty and I formatted it fresh this morning. Now the finder says 7 hours. Now this folder of DPX was the last exported to the Raid tower which is full so perhaps it's a less contiguous file and that is resulting in the 7 hour copy prediction. Not sure but sort of makes sense to me.
---In Avid-L2@yahoogroups.com, <bigfish@...> wrote :
Well that's what the finder projects and that seems to be about the time it takes on a virgin Hitachi NAS 4TB just formatted for three projects I have done. It's connected usb 3 being fed by a usb 3 connected Arcea Raid Tower with 8 drives. Perhaps the finder estimate isn't perfectly accurate but it does copy in that ball park. I think having a fresh format and a completely sequential copy helps. Also the source files were exported to the Raid tower in a sequential manner.
If the estimate is a bit rounded down based on the finder saying 5 hours and that could mean maybe 6 hours was more the actual time I can't say that I watched it that closely but it did copy pretty fast.
I did a similar copy yesterday to the new 20TB Graid and that said 5 hours to copy which would make more sense given it is a raid.
---In Avid-L2@yahoogroups.com, <bogdan_grigorescu@...> wrote :
From: "bigfish@... [Avid-L2]" <Avid-L2@yahoogroups.com>
To: Avid-L2@yahoogroups.com
Sent: Thursday, December 15, 2016 2:20 PM
Subject: Re: [Avid-L2] OT: Shotput Pro Copy/Process times?
---In Avid-L2@yahoogroups.com, <bouke@...> wrote :
If the estimate is a bit rounded down based on the finder saying 5 hours and that could mean maybe 6 hours was more the actual time I can't say that I watched it that closely but it did copy pretty fast.
I did a similar copy yesterday to the new 20TB Graid and that said 5 hours to copy which would make more sense given it is a raid.
---In Avid-L2@yahoogroups.com, <bogdan_grigorescu@...> wrote :
'3.3 Tb folder of a 10 bit RGB DPX sequence takes approx 5 hours to copy from a raid tower to a single 4TB 7200 rpm drive in a cradle'
Let me doubt that - as it would mean the single spindle has a sustained write rate of 180MBps, which I have yet to see.
BG
www.finale.tv
From: "bigfish@... [Avid-L2]" <Avid-L2@yahoogroups.com>
To: Avid-L2@yahoogroups.com
Sent: Thursday, December 15, 2016 2:20 PM
Subject: Re: [Avid-L2] OT: Shotput Pro Copy/Process times?
This is not for us this is a delivery requirement for the network. The network requires a copy of the Camera Original with a MD5 checksum log file. So far it doesn't seem like a 1TB folder of camera media should take anywhere near 12 or 24 hours.
As we are new to the software I'm wondering if there is anything they did in setting up the copy that could contribute to the lengthy time.
I know from my own experience with a 3.3 Tb folder of a 10 bit RGB DPX sequence takes approx 5 hours to copy from a raid tower to a single 4TB 7200 rpm drive in a cradle with everything connected to a usb 3 card.
The first thing they have tried to copy is about 1TB so it would seem something is messed up if it is projecting 12 to 24 hours to copy.
As we are new to the software I'm wondering if there is anything they did in setting up the copy that could contribute to the lengthy time.
I know from my own experience with a 3.3 Tb folder of a 10 bit RGB DPX sequence takes approx 5 hours to copy from a raid tower to a single 4TB 7200 rpm drive in a cradle with everything connected to a usb 3 card.
The first thing they have tried to copy is about 1TB so it would seem something is messed up if it is projecting 12 to 24 hours to copy.
---In Avid-L2@yahoogroups.com, <bouke@...> wrote :
MD5 is a very simple simple math operation, but it requires a total read of the source and target file.
So, you need to copy the file, and read both source and target, and do a bit of math. Depending on your system, this will take am much as 3 times more time as just a single copy.
Why are your doing this to yourself? One copy (Even with checksum) is not enough. You need at least 2 backups. So why bother?
Make TWO copies of the source, and you're as safe (Concidering a Md5 checksum good backup gets eaten by a dog. Wouldn't you like a non checksum copy?)
On Dec 15, 2016, at 21:10, Bogdan Grigorescu bogdan_grigorescu@... [Avid-L2] <Avid-L2@yahoogroups.com> wrote:how long for the copy if you turn off MD5?in my experience, MD5 verification essentially doubles the copy time.and DITs in the field normally use XXHash-64 checksum, which is significantly faster.BG
From: "John Moore bigfish@... [Avid-L2]" <Avid-L2@yahoogroups.com>
To: Yahoogroups <avid-l2@yahoogroups.com>; Yahoogroups <editing-list@yahoogroups.com>
Sent: Thursday, December 15, 2016 11:46 AM
Subject: [Avid-L2] OT: Shotput Pro Copy/Process times?After my recent thread regarding copying software the offered MD5 checksum and a log Shotput Pro was suggested by several. We purchased it and on a MacPro Tower with a 4TB Graid source drive and a 20TB Graid target drive, both connected to a usb 3 card for approximately 1 TB of Panasonic Varicam 35 4096x2160 avc-intra media the computer said 24 hours to copy. After a few restarts they moved the software to a different MacPro Tower. Both Macs are mid 2010 modified to 12 core running OS 10.9.5. The second computer said 12 hours but both seemed to stall in the process.We are not familiar with this process but I have been told by a more experienced friend the MD5 is CPU intensive. The CPU are 12 core 3.33Ghz. I can't believe that DITs in the field are taking 12 hours to copy drives to comply with the checksum MD5 log that is being required by the network. Most of our shoots have more like 2TB of camera media. I know there are options to just check file size and other check sum types that aren't as time consuming but the network specifically requests MD5. Is the large size of video files making the MD5 process much longer than it would be with more typical data that IT folks would deal with.What should 1TB of camera media take to copy on the above listed drives with an MD5 checksum.Also due to our OS they are running ShotPut Pro 5.x not 6 as that requires OS 10.10 which won't work with the rest of our infrastructure.John Moore Barking Trout Productions Studio City, CA bigfish@...
__._,_.___
Posted by: bigfish@pacbell.net
| Reply via web post | • | Reply to sender | • | Reply to group | • | Start a New Topic | • | Messages in this topic (14) |
Have you tried the highest rated email app?
With 4.5 stars in iTunes, the Yahoo Mail app is the highest rated email app on the market. What are you waiting for? Now you can access all your inboxes (Gmail, Outlook, AOL and more) in one place. Never delete an email again with 1000GB of free cloud storage.
this is the Avid-L2
.
__,_._,___
No comments:
Post a Comment